[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:41 dist-test-slave-vdtf krb5kdc[813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:41 dist-test-slave-vdtf krb5kdc[813](info): set up 2 sockets
Sep 25 22:45:41 dist-test-slave-vdtf krb5kdc[813](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 22:45:42.698313 806 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:45:42 dist-test-slave-vdtf krb5kdc[813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:45:42.820892 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40171
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:40171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:42.930914 826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:42.931164 826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:42.931203 826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:42.935135 826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:42.935201 826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:42.935221 826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:42.935238 826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:42.935253 826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:42.940495 826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:40171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:42.941534 826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:42.942592 826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:42.953218 831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:42.953243 834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:42.953328 826 server_base.cc:1034] running on GCE node
W20260925 22:45:42.953521 832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:42.954151 826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:42.955257 826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:42.956432 826 hybrid_clock.cc:648] HybridClock initialized: now 1790376342956365 us; error 79 us; skew 500 ppm
Sep 25 22:45:42 dist-test-slave-vdtf krb5kdc[813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:42.959239 826 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:45:42.960242 826 webserver.cc:460] Webserver started at http://127.0.201.190:34469/ using document root <none> and password file <none>
I20260925 22:45:42.960840 826 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:42.960904 826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:42.961110 826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:42.963184 826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "5b06288831c64a90b8e4429750e265bb"
format_stamp: "Formatted at 2026-09-25 22:45:42 on dist-test-slave-vdtf"
server_key: "052b5f4ace72993a5035865ffe0e95f1"
server_key_iv: "fdbc93eab69f72735a69368d5d207757"
server_key_version: "encryptionkey@0"
I20260925 22:45:42.963814 826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "5b06288831c64a90b8e4429750e265bb"
format_stamp: "Formatted at 2026-09-25 22:45:42 on dist-test-slave-vdtf"
server_key: "052b5f4ace72993a5035865ffe0e95f1"
server_key_iv: "fdbc93eab69f72735a69368d5d207757"
server_key_version: "encryptionkey@0"
I20260925 22:45:42.967597 826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:42.969990 841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:42.971238 826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:42.971362 826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "5b06288831c64a90b8e4429750e265bb"
format_stamp: "Formatted at 2026-09-25 22:45:42 on dist-test-slave-vdtf"
server_key: "052b5f4ace72993a5035865ffe0e95f1"
server_key_iv: "fdbc93eab69f72735a69368d5d207757"
server_key_version: "encryptionkey@0"
I20260925 22:45:42.971462 826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:42.991345 826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:42.994850 826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:42.995072 826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:43.002290 826 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:40171
I20260925 22:45:43.002295 893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:40171 every 8 connection(s)
I20260925 22:45:43.003307 826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:43.005976 894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:43.007773 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 826
I20260925 22:45:43.007926 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:45:43.009188 806 external_mini_cluster.cc:1428] Setting key 2f017560e458b3107a1fac75d424bfdb
I20260925 22:45:43.011713 894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb: Bootstrap starting.
I20260925 22:45:43.014343 894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:43.015050 894 log.cc:824] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:43.016954 894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb: No bootstrap required, opened a new log
I20260925 22:45:43.020337 894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b06288831c64a90b8e4429750e265bb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40171 } }
I20260925 22:45:43.020546 894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:43.020609 894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b06288831c64a90b8e4429750e265bb, State: Initialized, Role: FOLLOWER
I20260925 22:45:43.021018 894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b06288831c64a90b8e4429750e265bb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40171 } }
I20260925 22:45:43.021131 894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:43.021215 894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:43.021325 894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:43.022240 894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b06288831c64a90b8e4429750e265bb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40171 } }
I20260925 22:45:43.022560 894 leader_election.cc:302] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b06288831c64a90b8e4429750e265bb; no voters:
I20260925 22:45:43.022894 894 leader_election.cc:288] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:43.022948 899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:43.023124 899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [term 1 LEADER]: Becoming Leader. State: Replica: 5b06288831c64a90b8e4429750e265bb, State: Running, Role: LEADER
I20260925 22:45:43.023416 899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b06288831c64a90b8e4429750e265bb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40171 } }
I20260925 22:45:43.024108 894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:43.024956 900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b06288831c64a90b8e4429750e265bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b06288831c64a90b8e4429750e265bb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40171 } } }
I20260925 22:45:43.025096 900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:43.025436 901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b06288831c64a90b8e4429750e265bb. Latest consensus state: current_term: 1 leader_uuid: "5b06288831c64a90b8e4429750e265bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b06288831c64a90b8e4429750e265bb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40171 } } }
I20260925 22:45:43.025522 901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:43.025987 908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:43.029247 908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:43.035768 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:43.012401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.0.201.190:40171)
0925 22:45:43.012821 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:43.012843 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:43.013867 (+ 1024us) server_negotiation.cc:407] Connection header received
0925 22:45:43.017377 (+ 3510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:43.017394 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:43.017699 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:43.018015 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:43.020200 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.021456 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.022691 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.023299 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.024796 (+ 1497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:43.028671 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:43.028676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:43.028706 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:43.028738 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:43.031034 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.031641 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.031647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.031652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.031819 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.032216 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.032219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.032221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.032705 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:43.032882 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:43.034096 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0925 22:45:43.034769 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
I20260925 22:45:43.037235 908 catalog_manager.cc:1347] Generated new cluster ID: e095e2e92a19478d8de2f8692639ae35
I20260925 22:45:43.037302 908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:43.046336 908 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:43.047237 908 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:43.053325 908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b06288831c64a90b8e4429750e265bb: Generated new TSK 0
I20260925 22:45:43.054108 908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:45:43.091591 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40171
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:43.206845 920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:43.207182 920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:43.207228 920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:43.211516 920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:43.211586 920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:43.211683 920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:45:43.217206 920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40171
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:43.218273 920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:43.219292 920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:43.228964 925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:43.228969 928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:43.229065 920 server_base.cc:1034] running on GCE node
W20260925 22:45:43.228952 926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:43.229604 920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:43.230225 920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:43.231406 920 hybrid_clock.cc:648] HybridClock initialized: now 1790376343231358 us; error 62 us; skew 500 ppm
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:43.234227 920 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:45:43.235157 920 webserver.cc:460] Webserver started at http://127.0.201.129:40395/ using document root <none> and password file <none>
I20260925 22:45:43.235656 920 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:43.235777 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:43.235948 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:43.237929 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "517a094921374c61b962ef17f6116f30"
format_stamp: "Formatted at 2026-09-25 22:45:43 on dist-test-slave-vdtf"
server_key: "4f06b850c7b7ad3fea79037410b3f7fc"
server_key_iv: "e6c0469eff9dbc7aac3b6b6d93238e69"
server_key_version: "encryptionkey@0"
I20260925 22:45:43.238436 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "517a094921374c61b962ef17f6116f30"
format_stamp: "Formatted at 2026-09-25 22:45:43 on dist-test-slave-vdtf"
server_key: "4f06b850c7b7ad3fea79037410b3f7fc"
server_key_iv: "e6c0469eff9dbc7aac3b6b6d93238e69"
server_key_version: "encryptionkey@0"
I20260925 22:45:43.242022 920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 22:45:43.244383 935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:43.245558 920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:45:43.245651 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "517a094921374c61b962ef17f6116f30"
format_stamp: "Formatted at 2026-09-25 22:45:43 on dist-test-slave-vdtf"
server_key: "4f06b850c7b7ad3fea79037410b3f7fc"
server_key_iv: "e6c0469eff9dbc7aac3b6b6d93238e69"
server_key_version: "encryptionkey@0"
I20260925 22:45:43.245731 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:43.271875 920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:43.274715 920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:43.274864 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:43.275401 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:43.276292 920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:43.276337 920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:43.276376 920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:43.276397 920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:43.286696 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:40463
I20260925 22:45:43.286715 1048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:40463 every 8 connection(s)
I20260925 22:45:43.287593 920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:43.288255 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 920
I20260925 22:45:43.288370 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:43.288640 806 external_mini_cluster.cc:1428] Setting key 652c927aed9d8715c053295e3a99ddd6
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:45:43.302426 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:43.289565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:33875 (local address 127.0.201.190:40171)
0925 22:45:43.289700 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:43.289704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:43.290588 (+ 884us) server_negotiation.cc:407] Connection header received
0925 22:45:43.292012 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:43.292017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:43.292075 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:43.292172 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:43.293305 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.294234 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.295408 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.295846 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.295961 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:43.298555 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:43.298561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:43.298567 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:43.298592 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:43.300302 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.300910 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.300912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.300913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.300987 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.301361 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.301363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.301364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.301532 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:43.301642 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:43.302133 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 22:45:43.302256 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:45:43.303221 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:43.289972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40171 (local address 127.0.201.129:33875)
0925 22:45:43.290419 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:43.290483 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:43.291806 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:43.292320 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:43.292326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:43.292668 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:43.293158 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:43.293171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.294387 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.294389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:43.295282 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:43.295291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.295984 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.295987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:43.296123 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:43.296717 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:43.296733 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:43.298368 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:43.300426 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:43.300432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:43.300449 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:43.300814 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:43.301079 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:43.301085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:43.301090 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:43.301270 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:43.301644 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:43.301653 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:43.301885 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 22:45:43.302078 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 22:45:43.302389 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":121,"threads_started":1}
I20260925 22:45:43.304507 1049 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40171
I20260925 22:45:43.304773 1049 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:43.305286 1049 heartbeater.cc:511] Master 127.0.201.190:40171 requested a full tablet report, sending...
I20260925 22:45:43.306948 858 ts_manager.cc:194] Registered new tserver with Master: 517a094921374c61b962ef17f6116f30 (127.0.201.129:40463)
I20260925 22:45:43.308641 858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:33875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:45:43.318491 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40171
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:43.428191 1054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:43.428433 1054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:43.428478 1054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:43.432371 1054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:43.432420 1054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:43.432475 1054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:45:43.437475 1054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40171
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:43.438378 1054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:43.439388 1054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:43.448482 1060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:43.448470 1059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:43.448467 1062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:43.448563 1054 server_base.cc:1034] running on GCE node
I20260925 22:45:43.449071 1054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:43.449704 1054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:43.450863 1054 hybrid_clock.cc:648] HybridClock initialized: now 1790376343450835 us; error 39 us; skew 500 ppm
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:43.453521 1054 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:45:43.454453 1054 webserver.cc:460] Webserver started at http://127.0.201.130:43601/ using document root <none> and password file <none>
I20260925 22:45:43.455008 1054 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:43.455096 1054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:43.455312 1054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:43.457338 1054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "af1b227445cd47859203ac4ef39aadb7"
format_stamp: "Formatted at 2026-09-25 22:45:43 on dist-test-slave-vdtf"
server_key: "ecd45a3281b8ece75aa22dbdce9ac2d7"
server_key_iv: "83c99cf98e0b70c614f44161bf460b35"
server_key_version: "encryptionkey@0"
I20260925 22:45:43.457904 1054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "af1b227445cd47859203ac4ef39aadb7"
format_stamp: "Formatted at 2026-09-25 22:45:43 on dist-test-slave-vdtf"
server_key: "ecd45a3281b8ece75aa22dbdce9ac2d7"
server_key_iv: "83c99cf98e0b70c614f44161bf460b35"
server_key_version: "encryptionkey@0"
I20260925 22:45:43.461572 1054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:45:43.464026 1069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:43.465209 1054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:45:43.465344 1054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "af1b227445cd47859203ac4ef39aadb7"
format_stamp: "Formatted at 2026-09-25 22:45:43 on dist-test-slave-vdtf"
server_key: "ecd45a3281b8ece75aa22dbdce9ac2d7"
server_key_iv: "83c99cf98e0b70c614f44161bf460b35"
server_key_version: "encryptionkey@0"
I20260925 22:45:43.465451 1054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:43.480947 1054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:43.483641 1054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:43.483873 1054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:43.484429 1054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:43.485225 1054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:43.485294 1054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:43.485363 1054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:43.485414 1054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:43.495211 1054 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:34417
I20260925 22:45:43.495231 1182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:34417 every 8 connection(s)
I20260925 22:45:43.496122 1054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:43.504637 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 1054
I20260925 22:45:43.504777 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:43.505082 806 external_mini_cluster.cc:1428] Setting key c6fe7018ab92c6cd70880797e4b0e8fd
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:43.510769 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:43.497951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:46895 (local address 127.0.201.190:40171)
0925 22:45:43.498101 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:43.498105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:43.499788 (+ 1683us) server_negotiation.cc:407] Connection header received
0925 22:45:43.500726 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:43.500729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:43.500790 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:43.500899 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:43.502103 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.502620 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.503787 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.504018 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.504132 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:43.506816 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:43.506820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:43.506825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:43.506853 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:43.508501 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.509130 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.509133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.509134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.509210 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.509590 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.509596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.509601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.509855 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:43.509963 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:43.510419 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 22:45:43.510560 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:45:43.511467 1185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:43.498340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40171 (local address 127.0.201.130:46895)
0925 22:45:43.499198 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:43.499241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:43.500484 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:43.501109 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:43.501116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:43.501469 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:43.501946 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:43.501959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.502762 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.502765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:43.503640 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:43.503648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.504168 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.504173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:43.504337 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:43.504979 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:43.504995 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:43.506655 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:43.508610 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:43.508620 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:43.508636 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:43.509027 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:43.509298 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:43.509302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:43.509307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:43.509487 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:43.509957 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:43.509961 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:43.510184 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 22:45:43.510333 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 22:45:43.510569 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":524,"threads_started":1}
I20260925 22:45:43.512640 1183 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40171
I20260925 22:45:43.512851 1183 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:43.513314 1183 heartbeater.cc:511] Master 127.0.201.190:40171 requested a full tablet report, sending...
I20260925 22:45:43.514321 858 ts_manager.cc:194] Registered new tserver with Master: af1b227445cd47859203ac4ef39aadb7 (127.0.201.130:34417)
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:45:43.514926 858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:46895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:45:43.531095 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40171
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:43.646379 1188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:43.646634 1188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:43.646679 1188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:43.650928 1188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:43.651026 1188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:43.651091 1188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:45:43.656420 1188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40171
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:43.657403 1188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:43.658329 1188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:43.667538 1193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:43.667481 1196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:43.667600 1188 server_base.cc:1034] running on GCE node
W20260925 22:45:43.667466 1194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:43.668164 1188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:43.668828 1188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:43.670037 1188 hybrid_clock.cc:648] HybridClock initialized: now 1790376343670007 us; error 45 us; skew 500 ppm
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:43.672847 1188 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:45:43.673681 1188 webserver.cc:460] Webserver started at http://127.0.201.131:46787/ using document root <none> and password file <none>
I20260925 22:45:43.674180 1188 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:43.674235 1188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:43.674389 1188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:43.676576 1188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "9d6beb2a754543fbac0a99645d3b9eab"
format_stamp: "Formatted at 2026-09-25 22:45:43 on dist-test-slave-vdtf"
server_key: "9a0a1bf37ea574d22cae3ea0478439d5"
server_key_iv: "30b15199e70c1e1c270b5cc3b9b05df3"
server_key_version: "encryptionkey@0"
I20260925 22:45:43.677073 1188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "9d6beb2a754543fbac0a99645d3b9eab"
format_stamp: "Formatted at 2026-09-25 22:45:43 on dist-test-slave-vdtf"
server_key: "9a0a1bf37ea574d22cae3ea0478439d5"
server_key_iv: "30b15199e70c1e1c270b5cc3b9b05df3"
server_key_version: "encryptionkey@0"
I20260925 22:45:43.680696 1188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:45:43.683032 1203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:43.684291 1188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:45:43.684389 1188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "9d6beb2a754543fbac0a99645d3b9eab"
format_stamp: "Formatted at 2026-09-25 22:45:43 on dist-test-slave-vdtf"
server_key: "9a0a1bf37ea574d22cae3ea0478439d5"
server_key_iv: "30b15199e70c1e1c270b5cc3b9b05df3"
server_key_version: "encryptionkey@0"
I20260925 22:45:43.684470 1188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:43.716151 1188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:43.719043 1188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:43.719190 1188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:43.719715 1188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:43.720470 1188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:43.720521 1188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:43.720562 1188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:43.720580 1188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:43.730357 1188 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:35515
I20260925 22:45:43.730384 1316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:35515 every 8 connection(s)
I20260925 22:45:43.731200 1188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:43.737519 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 1188
I20260925 22:45:43.737635 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:43.737883 806 external_mini_cluster.cc:1428] Setting key b02031d9548f5ef80684148a6dae13ff
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:43.744426 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:43.732980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:33929 (local address 127.0.201.190:40171)
0925 22:45:43.733115 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:43.733119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:43.733918 (+ 799us) server_negotiation.cc:407] Connection header received
0925 22:45:43.735186 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:43.735189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:43.735234 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:43.735309 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:43.736497 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.737030 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.738223 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.738492 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.738575 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:43.741314 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:43.741317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:43.741319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:43.741341 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:43.742642 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.743089 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.743091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.743093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.743169 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.743443 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.743445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.743447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.743599 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:43.743669 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:43.744212 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 22:45:43.744305 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:45:43.745173 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:43.733365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40171 (local address 127.0.201.131:33929)
0925 22:45:43.733766 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:43.733808 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:43.734984 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:43.735466 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:43.735472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:43.735898 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:43.736378 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:43.736391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.737133 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.737135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:43.738109 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:43.738118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.738628 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.738633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:43.738853 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:43.739482 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:43.739501 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:43.741155 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:43.742740 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:43.742744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:43.742755 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:43.742994 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:43.743244 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:43.743246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:43.743247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:43.743367 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:43.743720 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:43.743728 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:43.743983 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 22:45:43.744159 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 22:45:43.744386 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260925 22:45:43.746340 1317 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40171
I20260925 22:45:43.746556 1317 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:43.747021 1317 heartbeater.cc:511] Master 127.0.201.190:40171 requested a full tablet report, sending...
I20260925 22:45:43.748040 858 ts_manager.cc:194] Registered new tserver with Master: 9d6beb2a754543fbac0a99645d3b9eab (127.0.201.131:35515)
I20260925 22:45:43.748582 858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:33929
I20260925 22:45:43.752094 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:45:43.777390 1324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:43.768002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.0.201.129:40463)
0925 22:45:43.768373 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:43.768386 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:43.768411 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:45:43.768507 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:43.768514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:43.768709 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:43.768832 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:43.769502 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.770314 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.771091 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.771491 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.771577 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:43.774002 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:43.774015 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:43.774028 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:43.774051 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:43.775594 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.776058 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.776063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.776064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.776140 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.776404 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.776407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.776408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.776645 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:43.776810 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:43.777040 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 22:45:43.777201 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
W20260925 22:45:43.778393 963 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51500
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:45:43.803355 1324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:43.795553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.0.201.129:40463)
0925 22:45:43.795758 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:43.795762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:43.795810 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:45:43.795927 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:43.795930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:43.795972 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:43.796042 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:43.796684 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.797170 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.798022 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:43.798232 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:43.798304 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:43.800676 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:43.800679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:43.800681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:43.800702 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:43.801898 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.802244 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.802246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.802247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.802323 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:43.802587 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:43.802589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:43.802590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:43.802730 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:43.802835 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:43.803051 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 22:45:43.803141 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:45:43.804539 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 920
I20260925 22:45:43.811163 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 1054
I20260925 22:45:43.817364 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 1188
I20260925 22:45:43.823309 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 826
2026-09-25T22:45:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2203 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[1331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[1331](info): set up 2 sockets
Sep 25 22:45:43 dist-test-slave-vdtf krb5kdc[1331](info): commencing operation
krb5kdc: starting...
W20260925 22:45:44.859361 806 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:45:44 dist-test-slave-vdtf krb5kdc[1331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:45:44.978192 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:44579
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:44579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:45.084434 1344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:45.084714 1344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:45.084785 1344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:45.088795 1344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:45.088889 1344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:45.088923 1344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:45.088981 1344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:45.089035 1344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:45.094794 1344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:44579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:45.095989 1344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:45.097023 1344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:45.105540 1350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:45.105559 1349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:45.105552 1352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:45.105616 1344 server_base.cc:1034] running on GCE node
I20260925 22:45:45.106305 1344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:45.107409 1344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:45.108606 1344 hybrid_clock.cc:648] HybridClock initialized: now 1790376345108575 us; error 43 us; skew 500 ppm
Sep 25 22:45:45 dist-test-slave-vdtf krb5kdc[1331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:45.122416 1344 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:45:45.123510 1344 webserver.cc:460] Webserver started at http://127.0.201.190:33183/ using document root <none> and password file <none>
I20260925 22:45:45.124150 1344 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:45.124241 1344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:45.124464 1344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:45.126513 1344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "f68a717e0fc446d4a1bb2ab778825f8d"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "1f56baf523518f04a3b8865742d8a9f0"
server_key_iv: "546e052aef2354c3465cec58bf8e020e"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.127063 1344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "f68a717e0fc446d4a1bb2ab778825f8d"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "1f56baf523518f04a3b8865742d8a9f0"
server_key_iv: "546e052aef2354c3465cec58bf8e020e"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.130656 1344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:45:45.132957 1359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:45.134222 1344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:45:45.134370 1344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "f68a717e0fc446d4a1bb2ab778825f8d"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "1f56baf523518f04a3b8865742d8a9f0"
server_key_iv: "546e052aef2354c3465cec58bf8e020e"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.134476 1344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:45.144930 1344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:45.148082 1344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:45.148268 1344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:45.154896 1344 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:44579
I20260925 22:45:45.154924 1411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:44579 every 8 connection(s)
I20260925 22:45:45.155797 1344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:45.158381 1412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:45.163757 1412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d: Bootstrap starting.
I20260925 22:45:45.164193 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 1344
I20260925 22:45:45.164278 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:45:45.164531 806 external_mini_cluster.cc:1428] Setting key 357c90df097ba52e8992ac7d68f283da
I20260925 22:45:45.166425 1412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:45.167290 1412 log.cc:824] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:45.169484 1412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d: No bootstrap required, opened a new log
I20260925 22:45:45.173328 1412 raft_consensus.cc:348] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68a717e0fc446d4a1bb2ab778825f8d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44579 } }
I20260925 22:45:45.173579 1412 raft_consensus.cc:374] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:45.173659 1412 raft_consensus.cc:729] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68a717e0fc446d4a1bb2ab778825f8d, State: Initialized, Role: FOLLOWER
Sep 25 22:45:45 dist-test-slave-vdtf krb5kdc[1331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:45.174144 1412 consensus_queue.cc:260] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68a717e0fc446d4a1bb2ab778825f8d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44579 } }
I20260925 22:45:45.174304 1412 raft_consensus.cc:388] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:45.174432 1412 raft_consensus.cc:482] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:45.174551 1412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:45.175832 1412 raft_consensus.cc:504] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68a717e0fc446d4a1bb2ab778825f8d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44579 } }
I20260925 22:45:45.176189 1412 leader_election.cc:302] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f68a717e0fc446d4a1bb2ab778825f8d; no voters:
I20260925 22:45:45.176520 1412 leader_election.cc:288] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:45.176606 1417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:45.176815 1417 raft_consensus.cc:686] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [term 1 LEADER]: Becoming Leader. State: Replica: f68a717e0fc446d4a1bb2ab778825f8d, State: Running, Role: LEADER
I20260925 22:45:45.177220 1417 consensus_queue.cc:237] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68a717e0fc446d4a1bb2ab778825f8d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44579 } }
I20260925 22:45:45.177834 1412 sys_catalog.cc:564] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:45.179014 1418 sys_catalog.cc:455] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f68a717e0fc446d4a1bb2ab778825f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68a717e0fc446d4a1bb2ab778825f8d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44579 } } }
I20260925 22:45:45.179023 1419 sys_catalog.cc:455] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f68a717e0fc446d4a1bb2ab778825f8d. Latest consensus state: current_term: 1 leader_uuid: "f68a717e0fc446d4a1bb2ab778825f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68a717e0fc446d4a1bb2ab778825f8d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44579 } } }
I20260925 22:45:45.179172 1418 sys_catalog.cc:458] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:45.179173 1419 sys_catalog.cc:458] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:45.179922 1426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:45.180083 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.165972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.0.201.190:44579)
0925 22:45:45.166407 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:45.166426 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:45.166470 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:45:45.167332 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:45.167357 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:45.167795 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:45.168119 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:45.168750 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.170179 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.171005 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.171718 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.171876 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.174460 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:45.174467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:45.174484 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:45.174515 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:45.176529 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:45.177122 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:45.177127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:45.177133 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:45.177270 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:45.177592 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:45.177596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:45.177597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:45.178124 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:45.178310 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:45.178646 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 22:45:45.178976 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":113,"threads_started":1}
I20260925 22:45:45.183317 1426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:45.188683 1426 catalog_manager.cc:1347] Generated new cluster ID: 61ac1265ab994b0ab687388c11101c96
I20260925 22:45:45.188747 1426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:45.204557 1426 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:45.205799 1426 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:45.211882 1426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f68a717e0fc446d4a1bb2ab778825f8d: Generated new TSK 0
I20260925 22:45:45.212657 1426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:45:45.239306 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44579
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:45.347791 1438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:45.348027 1438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:45.348067 1438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:45.351895 1438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:45.351940 1438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:45.351995 1438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:45:45.357052 1438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:45.357928 1438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:45.358834 1438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:45.367879 1444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:45.367863 1443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:45.368073 1446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:45.368041 1438 server_base.cc:1034] running on GCE node
I20260925 22:45:45.368533 1438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:45.369127 1438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:45.370314 1438 hybrid_clock.cc:648] HybridClock initialized: now 1790376345370296 us; error 34 us; skew 500 ppm
Sep 25 22:45:45 dist-test-slave-vdtf krb5kdc[1331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:45.372987 1438 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:45:45.373898 1438 webserver.cc:460] Webserver started at http://127.0.201.129:36813/ using document root <none> and password file <none>
I20260925 22:45:45.374455 1438 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:45.374554 1438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:45.374778 1438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:45.376842 1438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "a65765d29ea642eeb39dc7162141cd1f"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "b1a206e0e7be9e44fdb032f02f25b346"
server_key_iv: "394601e0e1e370b07c6ab80dd41197b0"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.377399 1438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "a65765d29ea642eeb39dc7162141cd1f"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "b1a206e0e7be9e44fdb032f02f25b346"
server_key_iv: "394601e0e1e370b07c6ab80dd41197b0"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.381213 1438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:45:45.383523 1453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:45.384729 1438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:45:45.384859 1438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "a65765d29ea642eeb39dc7162141cd1f"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "b1a206e0e7be9e44fdb032f02f25b346"
server_key_iv: "394601e0e1e370b07c6ab80dd41197b0"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.384968 1438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:45.413035 1438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:45.416484 1438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:45.416712 1438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:45.417341 1438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:45.418208 1438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:45.418282 1438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:45.418355 1438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:45.418401 1438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:45.427052 1438 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:46395
I20260925 22:45:45.427069 1566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:46395 every 8 connection(s)
I20260925 22:45:45.428072 1438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:45.435240 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 1438
I20260925 22:45:45.435382 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:45.435657 806 external_mini_cluster.cc:1428] Setting key 9b882ccacd94b46ed79a18da050f996c
Sep 25 22:45:45 dist-test-slave-vdtf krb5kdc[1331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:45.443014 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.429946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:33355 (local address 127.0.201.190:44579)
0925 22:45:45.430083 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:45.430087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:45.430926 (+ 839us) server_negotiation.cc:407] Connection header received
0925 22:45:45.432353 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:45.432356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:45.432416 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:45.432516 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:45.433674 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.434197 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.435325 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.435650 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.435781 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.438923 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:45.438926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:45.438928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:45.438951 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:45.440721 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:45.441226 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:45.441232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:45.441237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:45.441330 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:45.441746 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:45.441751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:45.441756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:45.442012 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:45.442131 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:45.442629 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 22:45:45.442772 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:45:45.443639 1569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.430264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44579 (local address 127.0.201.129:33355)
0925 22:45:45.430764 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:45.430808 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:45.432132 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:45.432693 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:45.432699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:45.433047 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:45.433518 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.433531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.434326 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.434334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.435201 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.435210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.435842 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.435846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.436029 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.436745 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:45.436764 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:45.438748 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:45.440844 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:45.440850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:45.440861 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:45.441114 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:45.441446 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:45.441449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:45.441450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:45.441605 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:45.442132 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:45.442137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:45.442374 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 22:45:45.442534 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 22:45:45.442778 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260925 22:45:45.444851 1567 heartbeater.cc:347] Connected to a master server at 127.0.201.190:44579
I20260925 22:45:45.445106 1567 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:45.445573 1567 heartbeater.cc:511] Master 127.0.201.190:44579 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:45:45.447203 1376 ts_manager.cc:194] Registered new tserver with Master: a65765d29ea642eeb39dc7162141cd1f (127.0.201.129:46395)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:45:45.448578 1376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:33355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:45:45.462744 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44579
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:45.572463 1572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:45.572759 1572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:45.572844 1572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:45.576922 1572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:45.577008 1572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:45.577095 1572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:45:45.582558 1572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:45.583642 1572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:45.584738 1572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:45.594295 1578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:45.594333 1577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:45.594313 1580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:45.594525 1572 server_base.cc:1034] running on GCE node
I20260925 22:45:45.594897 1572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:45.595539 1572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:45.596724 1572 hybrid_clock.cc:648] HybridClock initialized: now 1790376345596710 us; error 25 us; skew 500 ppm
Sep 25 22:45:45 dist-test-slave-vdtf krb5kdc[1331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:45.599411 1572 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:45:45.600373 1572 webserver.cc:460] Webserver started at http://127.0.201.130:38291/ using document root <none> and password file <none>
I20260925 22:45:45.600937 1572 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:45.601029 1572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:45.601271 1572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:45.603235 1572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "c494acb8576b4389a01bedd20482bdbb"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "e63dbdb6246b34a2085fba1d3e538ab1"
server_key_iv: "72c92b2b477d1b8eddf437639ae917bc"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.603850 1572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "c494acb8576b4389a01bedd20482bdbb"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "e63dbdb6246b34a2085fba1d3e538ab1"
server_key_iv: "72c92b2b477d1b8eddf437639ae917bc"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.607465 1572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:45.609867 1587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:45.611028 1572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:45.611159 1572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "c494acb8576b4389a01bedd20482bdbb"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "e63dbdb6246b34a2085fba1d3e538ab1"
server_key_iv: "72c92b2b477d1b8eddf437639ae917bc"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.611263 1572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:45.634220 1572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:45.637136 1572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:45.637354 1572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:45.637936 1572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:45.638720 1572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:45.638787 1572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:45.638855 1572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:45.638909 1572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:45.647848 1572 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:34085
I20260925 22:45:45.647881 1700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:34085 every 8 connection(s)
I20260925 22:45:45.648766 1572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
Sep 25 22:45:45 dist-test-slave-vdtf krb5kdc[1331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:45.658761 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 1572
I20260925 22:45:45.658892 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:45.659157 806 external_mini_cluster.cc:1428] Setting key cc17979c0e411e88227590371479a09b
I20260925 22:45:45.662320 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.650652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:53271 (local address 127.0.201.190:44579)
0925 22:45:45.650805 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:45.650810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:45.651702 (+ 892us) server_negotiation.cc:407] Connection header received
0925 22:45:45.653025 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:45.653027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:45.653074 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:45.653150 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:45.654255 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.654757 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.655899 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.656101 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.656176 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.658510 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:45.658513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:45.658515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:45.658564 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:45.660148 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:45.660748 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:45.660751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:45.660753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:45.660842 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:45.661156 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:45.661159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:45.661161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:45.661406 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:45.661515 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:45.661946 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 22:45:45.662085 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:45:45.663137 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.651112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44579 (local address 127.0.201.130:53271)
0925 22:45:45.651544 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:45.651586 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:45.652827 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:45.653303 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:45.653310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:45.653660 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:45.654132 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.654145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.654862 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.654864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.655802 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.655810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.656221 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.656224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.656401 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.656949 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:45.656965 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:45.658333 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:45.660259 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:45.660264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:45.660277 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:45.660637 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:45.660933 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:45.660935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:45.660936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:45.661056 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:45.661508 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:45.661513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:45.661720 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 22:45:45.661869 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 22:45:45.662102 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260925 22:45:45.664322 1701 heartbeater.cc:347] Connected to a master server at 127.0.201.190:44579
I20260925 22:45:45.664579 1701 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:45.665031 1701 heartbeater.cc:511] Master 127.0.201.190:44579 requested a full tablet report, sending...
I20260925 22:45:45.666041 1376 ts_manager.cc:194] Registered new tserver with Master: c494acb8576b4389a01bedd20482bdbb (127.0.201.130:34085)
I20260925 22:45:45.666589 1376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:53271
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:45:45.691816 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44579
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:45.803954 1706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:45.804220 1706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:45.804266 1706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:45.808940 1706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:45.809006 1706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:45.809098 1706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:45:45.814476 1706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:45.815613 1706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:45.816635 1706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:45.826280 1711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:45.826269 1714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:45.826277 1712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:45.826315 1706 server_base.cc:1034] running on GCE node
I20260925 22:45:45.826934 1706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:45.827551 1706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:45.828714 1706 hybrid_clock.cc:648] HybridClock initialized: now 1790376345828697 us; error 30 us; skew 500 ppm
Sep 25 22:45:45 dist-test-slave-vdtf krb5kdc[1331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:45.831928 1706 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:45:45.833076 1706 webserver.cc:460] Webserver started at http://127.0.201.131:38473/ using document root <none> and password file <none>
I20260925 22:45:45.833626 1706 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:45.833706 1706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:45.833909 1706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:45.836459 1706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "2166f0ae10c64427a89d421de253bfd1"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "ca0f7622f9046ff212204bba569f4bbb"
server_key_iv: "b2279ddc36c3afcec9e0b8edbd7fa467"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.837196 1706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "2166f0ae10c64427a89d421de253bfd1"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "ca0f7622f9046ff212204bba569f4bbb"
server_key_iv: "b2279ddc36c3afcec9e0b8edbd7fa467"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.842243 1706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 22:45:45.845062 1721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:45.846278 1706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:45:45.846426 1706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "2166f0ae10c64427a89d421de253bfd1"
format_stamp: "Formatted at 2026-09-25 22:45:45 on dist-test-slave-vdtf"
server_key: "ca0f7622f9046ff212204bba569f4bbb"
server_key_iv: "b2279ddc36c3afcec9e0b8edbd7fa467"
server_key_version: "encryptionkey@0"
I20260925 22:45:45.846535 1706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:45.874902 1706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:45.877866 1706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:45.878067 1706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:45.878741 1706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:45.879550 1706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:45.879619 1706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:45.879709 1706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:45.879741 1706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:45.888628 1706 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:41743
I20260925 22:45:45.888648 1834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:41743 every 8 connection(s)
I20260925 22:45:45.889571 1706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:45.898239 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 1706
I20260925 22:45:45.898376 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:45.898655 806 external_mini_cluster.cc:1428] Setting key e0255c08d32e45d8380a61907cb56191
Sep 25 22:45:45 dist-test-slave-vdtf krb5kdc[1331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:45.903159 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.891442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:51803 (local address 127.0.201.190:44579)
0925 22:45:45.891587 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:45.891590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:45.892412 (+ 822us) server_negotiation.cc:407] Connection header received
0925 22:45:45.893721 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:45.893724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:45.893777 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:45.893882 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:45.895014 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.895543 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.896700 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.896912 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.897021 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.899722 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:45.899726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:45.899728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:45.899766 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:45.901309 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:45.901758 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:45.901760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:45.901762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:45.901834 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:45.902156 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:45.902158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:45.902160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:45.902322 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:45.902400 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:45.902952 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 22:45:45.903045 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:45:45.904078 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.891856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44579 (local address 127.0.201.131:51803)
0925 22:45:45.892279 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:45.892319 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:45.893525 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:45.894064 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:45.894071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:45.894407 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:45.894873 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.894885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.895658 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.895662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.896571 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.896579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.897056 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.897060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.897220 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.897795 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:45.897811 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:45.899476 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:45.901412 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:45.901417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:45.901428 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:45.901675 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:45.901955 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:45.901957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:45.901958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:45.902082 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:45.902406 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:45.902411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:45.902724 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 22:45:45.902893 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 22:45:45.903136 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260925 22:45:45.905107 1835 heartbeater.cc:347] Connected to a master server at 127.0.201.190:44579
I20260925 22:45:45.905313 1835 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:45.905741 1835 heartbeater.cc:511] Master 127.0.201.190:44579 requested a full tablet report, sending...
I20260925 22:45:45.906688 1376 ts_manager.cc:194] Registered new tserver with Master: 2166f0ae10c64427a89d421de253bfd1 (127.0.201.131:41743)
I20260925 22:45:45.907222 1376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:51803
I20260925 22:45:45.912436 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:45:45.923628 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.916015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.0.201.190:44579)
0925 22:45:45.916141 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:45.916144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:45.916321 (+ 177us) server_negotiation.cc:407] Connection header received
0925 22:45:45.916434 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:45.916436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:45.916474 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:45.916540 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:45.917284 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.917801 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.918828 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.919048 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.919150 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.920707 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:45.920710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:45.920712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:45.920736 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:45.922105 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:45.922490 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:45.922492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:45.922493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:45.922563 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:45.922831 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:45.922834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:45.922835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:45.923054 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:45.923162 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:45.923406 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 22:45:45.923505 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:45:45.928340 1376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:45:45.931253 1376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:45.945489 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.939695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.0.201.129:46395)
0925 22:45:45.940151 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:45.940167 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:45.940192 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:45:45.940407 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:45.940414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:45.940621 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:45.940784 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:45.941295 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.942447 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.944185 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.944807 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.944939 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.945085 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:45.945242 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 22:45:45.945335 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":138,"threads_started":1}
I20260925 22:45:45.945482 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.939528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:46395 (local address 127.0.0.1:48066)
0925 22:45:45.940001 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:45.940047 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:45.940192 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:45.940983 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:45.940989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:45.941012 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:45.941166 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.941176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.942571 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.942574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.944054 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.944066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.944938 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.944949 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.945050 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.945085 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 22:45:45.945215 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 22:45:45.945335 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":92,"thread_start_us":79,"threads_started":1}
I20260925 22:45:45.945785 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.940295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:34085 (local address 127.0.0.1:43826)
0925 22:45:45.941019 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:45.941032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:45.941103 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:45.941764 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:45.941767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:45.941785 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:45.941869 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.941875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.943023 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.943030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.944114 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.944142 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.945372 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.945375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.945485 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.945518 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 22:45:45.945583 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 22:45:45.945665 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":56,"thread_start_us":89,"threads_started":1}
I20260925 22:45:45.946007 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.939669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:41743 (local address 127.0.0.1:43482)
0925 22:45:45.940461 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:45.940474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:45.940564 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:45.941221 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:45.941224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:45.941241 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:45.941347 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.941355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.942464 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.942469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.944088 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:45.944098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.945198 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.945201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:45.945286 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.945308 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:45:45.945361 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 22:45:45.945451 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":119,"thread_start_us":170,"threads_started":1}
I20260925 22:45:45.946265 1851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.940688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.0.201.130:34085)
0925 22:45:45.941337 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:45.941349 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:45.941368 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:45:45.941432 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:45.941437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:45.941614 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:45.941756 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:45.941988 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.942888 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.944243 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.945060 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.945175 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.945304 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:45.946006 (+ 702us) server_negotiation.cc:299] Negotiation successful
0925 22:45:45.946114 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":85,"threads_started":1}
I20260925 22:45:45.946705 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:45.940299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.0.201.131:41743)
0925 22:45:45.940795 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:45.940807 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:45.940828 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:45:45.940898 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:45.940903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:45.941093 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:45.941243 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:45.941467 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.942326 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.944215 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:45.945060 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:45.946221 (+ 1161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:45.946373 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:45.946469 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 22:45:45.946566 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":107,"threads_started":1}
I20260925 22:45:45.948817 1501 tablet_service.cc:1467] Processing CreateTablet for tablet 70a4ba61ed894faf89aebaf1b60d87ef (DEFAULT_TABLE table=test-table [id=6789d9d1ed8d4500b777a03eb8e8fcfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:45.949023 1635 tablet_service.cc:1467] Processing CreateTablet for tablet 70a4ba61ed894faf89aebaf1b60d87ef (DEFAULT_TABLE table=test-table [id=6789d9d1ed8d4500b777a03eb8e8fcfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:45.949265 1769 tablet_service.cc:1467] Processing CreateTablet for tablet 70a4ba61ed894faf89aebaf1b60d87ef (DEFAULT_TABLE table=test-table [id=6789d9d1ed8d4500b777a03eb8e8fcfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:45.950064 1635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70a4ba61ed894faf89aebaf1b60d87ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:45.950065 1769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70a4ba61ed894faf89aebaf1b60d87ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:45.950080 1501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70a4ba61ed894faf89aebaf1b60d87ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:45.956323 1852 tablet_bootstrap.cc:492] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f: Bootstrap starting.
I20260925 22:45:45.957691 1853 tablet_bootstrap.cc:492] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb: Bootstrap starting.
I20260925 22:45:45.958187 1854 tablet_bootstrap.cc:492] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1: Bootstrap starting.
I20260925 22:45:45.958673 1852 tablet_bootstrap.cc:654] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:45.959414 1852 log.cc:824] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:45.960307 1853 tablet_bootstrap.cc:654] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:45.960952 1854 tablet_bootstrap.cc:654] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:45.961201 1853 log.cc:824] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:45.961720 1854 log.cc:824] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:45.961889 1852 tablet_bootstrap.cc:492] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f: No bootstrap required, opened a new log
I20260925 22:45:45.962085 1852 ts_tablet_manager.cc:1397] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:45:45.963147 1853 tablet_bootstrap.cc:492] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb: No bootstrap required, opened a new log
I20260925 22:45:45.963318 1853 ts_tablet_manager.cc:1397] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:45:45.963584 1854 tablet_bootstrap.cc:492] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1: No bootstrap required, opened a new log
I20260925 22:45:45.963768 1854 ts_tablet_manager.cc:1397] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:45:45.965783 1852 raft_consensus.cc:348] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65765d29ea642eeb39dc7162141cd1f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46395 } } peers { permanent_uuid: "2166f0ae10c64427a89d421de253bfd1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 41743 } } peers { permanent_uuid: "c494acb8576b4389a01bedd20482bdbb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34085 } }
I20260925 22:45:45.966049 1852 raft_consensus.cc:374] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:45.966120 1852 raft_consensus.cc:729] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65765d29ea642eeb39dc7162141cd1f, State: Initialized, Role: FOLLOWER
I20260925 22:45:45.966638 1852 consensus_queue.cc:260] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65765d29ea642eeb39dc7162141cd1f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46395 } } peers { permanent_uuid: "2166f0ae10c64427a89d421de253bfd1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 41743 } } peers { permanent_uuid: "c494acb8576b4389a01bedd20482bdbb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34085 } }
I20260925 22:45:45.966995 1853 raft_consensus.cc:348] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65765d29ea642eeb39dc7162141cd1f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46395 } } peers { permanent_uuid: "2166f0ae10c64427a89d421de253bfd1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 41743 } } peers { permanent_uuid: "c494acb8576b4389a01bedd20482bdbb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34085 } }
I20260925 22:45:45.967239 1853 raft_consensus.cc:374] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:45.967291 1567 heartbeater.cc:503] Master 127.0.201.190:44579 was elected leader, sending a full tablet report...
I20260925 22:45:45.967304 1853 raft_consensus.cc:729] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c494acb8576b4389a01bedd20482bdbb, State: Initialized, Role: FOLLOWER
I20260925 22:45:45.967669 1852 ts_tablet_manager.cc:1428] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:45:45.967728 1853 consensus_queue.cc:260] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65765d29ea642eeb39dc7162141cd1f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46395 } } peers { permanent_uuid: "2166f0ae10c64427a89d421de253bfd1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 41743 } } peers { permanent_uuid: "c494acb8576b4389a01bedd20482bdbb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34085 } }
I20260925 22:45:45.968294 1854 raft_consensus.cc:348] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65765d29ea642eeb39dc7162141cd1f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46395 } } peers { permanent_uuid: "2166f0ae10c64427a89d421de253bfd1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 41743 } } peers { permanent_uuid: "c494acb8576b4389a01bedd20482bdbb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34085 } }
I20260925 22:45:45.968575 1854 raft_consensus.cc:374] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:45.968636 1854 raft_consensus.cc:729] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166f0ae10c64427a89d421de253bfd1, State: Initialized, Role: FOLLOWER
I20260925 22:45:45.968643 1853 ts_tablet_manager.cc:1428] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:45:45.968974 1701 heartbeater.cc:503] Master 127.0.201.190:44579 was elected leader, sending a full tablet report...
I20260925 22:45:45.969056 1854 consensus_queue.cc:260] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65765d29ea642eeb39dc7162141cd1f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46395 } } peers { permanent_uuid: "2166f0ae10c64427a89d421de253bfd1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 41743 } } peers { permanent_uuid: "c494acb8576b4389a01bedd20482bdbb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34085 } }
I20260925 22:45:45.969686 1835 heartbeater.cc:503] Master 127.0.201.190:44579 was elected leader, sending a full tablet report...
I20260925 22:45:45.969928 1854 ts_tablet_manager.cc:1428] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:45:46.122634 1859 raft_consensus.cc:482] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:46.122833 1859 raft_consensus.cc:504] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65765d29ea642eeb39dc7162141cd1f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46395 } } peers { permanent_uuid: "2166f0ae10c64427a89d421de253bfd1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 41743 } } peers { permanent_uuid: "c494acb8576b4389a01bedd20482bdbb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34085 } }
I20260925 22:45:46.124132 1859 leader_election.cc:288] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a65765d29ea642eeb39dc7162141cd1f (127.0.201.129:46395), 2166f0ae10c64427a89d421de253bfd1 (127.0.201.131:41743)
I20260925 22:45:46.128553 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:46.124365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:46395 (local address 127.0.201.130:46967)
0925 22:45:46.124514 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:46.124532 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:46.124629 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:46.124979 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:46.124981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:46.125029 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:46.125180 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:46.125187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.126170 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.126174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:46.127364 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:46.127373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.128122 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.128125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:46.128229 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:46.128255 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:45:46.128336 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:45:46.128413 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":67}
I20260925 22:45:46.128650 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:46.124448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:46967 (local address 127.0.201.129:46395)
0925 22:45:46.124585 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:46.124590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:46.124609 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:45:46.124780 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:46.124785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:46.124851 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:46.124943 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:46.125328 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.126026 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.127492 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.128002 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.128099 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:46.128140 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:46.128413 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 22:45:46.128502 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:45:46.128952 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:46.124661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:45197 (local address 127.0.201.131:41743)
0925 22:45:46.124852 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:46.124855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:46.125040 (+ 185us) server_negotiation.cc:407] Connection header received
0925 22:45:46.125167 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:46.125170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:46.125223 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:46.125301 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:46.125580 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.126254 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.127731 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.128234 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.128375 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:46.128418 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:46.128715 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 22:45:46.128802 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 22:45:46.129345 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:46.124559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:41743 (local address 127.0.201.130:45197)
0925 22:45:46.124929 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:46.124944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:46.125052 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:46.125327 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:46.125330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:46.125357 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:46.125466 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:46.125472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.126377 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.126380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:46.127559 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:46.127568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.128408 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.128412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:46.128538 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:46.128563 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:45:46.129126 (+ 563us) client_negotiation.cc:240] Negotiation successful
0925 22:45:46.129204 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":2,"spinlock_wait_cycles":5888,"thread_start_us":97,"threads_started":1}
I20260925 22:45:46.129364 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a4ba61ed894faf89aebaf1b60d87ef" candidate_uuid: "c494acb8576b4389a01bedd20482bdbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a65765d29ea642eeb39dc7162141cd1f" is_pre_election: true
I20260925 22:45:46.129714 1521 raft_consensus.cc:2454] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c494acb8576b4389a01bedd20482bdbb in term 0.
I20260925 22:45:46.130033 1789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a4ba61ed894faf89aebaf1b60d87ef" candidate_uuid: "c494acb8576b4389a01bedd20482bdbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166f0ae10c64427a89d421de253bfd1" is_pre_election: true
I20260925 22:45:46.130347 1789 raft_consensus.cc:2454] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c494acb8576b4389a01bedd20482bdbb in term 0.
I20260925 22:45:46.130312 1589 leader_election.cc:302] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a65765d29ea642eeb39dc7162141cd1f, c494acb8576b4389a01bedd20482bdbb; no voters:
I20260925 22:45:46.130600 1859 raft_consensus.cc:2781] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:46.130746 1859 raft_consensus.cc:482] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:46.130811 1859 raft_consensus.cc:3037] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:46.132030 1859 raft_consensus.cc:504] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65765d29ea642eeb39dc7162141cd1f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46395 } } peers { permanent_uuid: "2166f0ae10c64427a89d421de253bfd1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 41743 } } peers { permanent_uuid: "c494acb8576b4389a01bedd20482bdbb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34085 } }
I20260925 22:45:46.132354 1859 leader_election.cc:288] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [CANDIDATE]: Term 1 election: Requested vote from peers a65765d29ea642eeb39dc7162141cd1f (127.0.201.129:46395), 2166f0ae10c64427a89d421de253bfd1 (127.0.201.131:41743)
I20260925 22:45:46.132712 1789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a4ba61ed894faf89aebaf1b60d87ef" candidate_uuid: "c494acb8576b4389a01bedd20482bdbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166f0ae10c64427a89d421de253bfd1"
I20260925 22:45:46.132717 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a4ba61ed894faf89aebaf1b60d87ef" candidate_uuid: "c494acb8576b4389a01bedd20482bdbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a65765d29ea642eeb39dc7162141cd1f"
I20260925 22:45:46.132836 1521 raft_consensus.cc:3037] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:46.132835 1789 raft_consensus.cc:3037] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:46.134091 1521 raft_consensus.cc:2454] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c494acb8576b4389a01bedd20482bdbb in term 1.
I20260925 22:45:46.134091 1789 raft_consensus.cc:2454] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c494acb8576b4389a01bedd20482bdbb in term 1.
I20260925 22:45:46.134423 1588 leader_election.cc:302] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2166f0ae10c64427a89d421de253bfd1, a65765d29ea642eeb39dc7162141cd1f, c494acb8576b4389a01bedd20482bdbb; no voters:
I20260925 22:45:46.134598 1859 raft_consensus.cc:2781] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:46.134819 1859 raft_consensus.cc:686] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [term 1 LEADER]: Becoming Leader. State: Replica: c494acb8576b4389a01bedd20482bdbb, State: Running, Role: LEADER
I20260925 22:45:46.135149 1859 consensus_queue.cc:237] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65765d29ea642eeb39dc7162141cd1f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46395 } } peers { permanent_uuid: "2166f0ae10c64427a89d421de253bfd1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 41743 } } peers { permanent_uuid: "c494acb8576b4389a01bedd20482bdbb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34085 } }
I20260925 22:45:46.138489 1374 catalog_manager.cc:5495] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb reported cstate change: term changed from 0 to 1, leader changed from <none> to c494acb8576b4389a01bedd20482bdbb (127.0.201.130). New cstate: current_term: 1 leader_uuid: "c494acb8576b4389a01bedd20482bdbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65765d29ea642eeb39dc7162141cd1f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166f0ae10c64427a89d421de253bfd1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 41743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c494acb8576b4389a01bedd20482bdbb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34085 } health_report { overall_health: HEALTHY } } }
W20260925 22:45:46.140558 1836 tablet.cc:2369] T 70a4ba61ed894faf89aebaf1b60d87ef P 2166f0ae10c64427a89d421de253bfd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:45:46.149845 1702 tablet.cc:2369] T 70a4ba61ed894faf89aebaf1b60d87ef P c494acb8576b4389a01bedd20482bdbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:45:46.179397 1568 tablet.cc:2369] T 70a4ba61ed894faf89aebaf1b60d87ef P a65765d29ea642eeb39dc7162141cd1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 22:45:46 dist-test-slave-vdtf krb5kdc[1331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:45:46.186969 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:46.178251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.0.201.129:46395)
0925 22:45:46.178433 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:46.178437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:46.178632 (+ 195us) server_negotiation.cc:407] Connection header received
0925 22:45:46.178728 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:46.178730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:46.178771 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:46.178841 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:46.179628 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.180314 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.181126 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.181335 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.181422 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:46.183911 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:46.183925 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:46.183940 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:46.183962 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:46.185445 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:46.185835 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:46.185840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:46.185841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:46.185915 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:46.186145 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:46.186147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:46.186148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:46.186367 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:46.186526 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:46.186667 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 22:45:46.186758 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 22:45:46 dist-test-slave-vdtf krb5kdc[1331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376346, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:45:46 dist-test-slave-vdtf krb5kdc[1331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376346, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:45:46.213263 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:46.204827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.0.201.129:46395)
0925 22:45:46.204985 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:46.204989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:46.205149 (+ 160us) server_negotiation.cc:407] Connection header received
0925 22:45:46.205323 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:46.205326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:46.205368 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:46.205459 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:46.206120 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.206735 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.207567 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.207869 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.207992 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:46.210288 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:46.210292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:46.210294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:46.210320 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:46.211534 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:46.211991 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:46.211995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:46.211996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:46.212080 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:46.212360 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:46.212364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:46.212366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:46.212573 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:46.212706 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:46.212984 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 22:45:46.213103 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:45:46 dist-test-slave-vdtf krb5kdc[1331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:45:46 dist-test-slave-vdtf krb5kdc[1331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:45:46.238883 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:46.230266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.0.201.129:46395)
0925 22:45:46.230453 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:46.230457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:46.230623 (+ 166us) server_negotiation.cc:407] Connection header received
0925 22:45:46.230799 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:46.230803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:46.230851 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:46.230948 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:46.231858 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.232372 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.233226 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:46.233449 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:46.233569 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:46.236051 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:46.236055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:46.236058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:46.236094 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:46.237230 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:46.237647 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:46.237653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:46.237655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:46.237775 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:46.238064 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:46.238067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:46.238069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:46.238226 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:46.238364 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:46.238604 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 22:45:46.238719 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 22:45:46.240304 1481 tablet_service.cc:2908] Rejecting scan request for tablet 70a4ba61ed894faf89aebaf1b60d87ef: Uninitialized: clean time has not yet been initialized
I20260925 22:45:46.242353 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 1438
I20260925 22:45:46.248972 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 1572
I20260925 22:45:46.254487 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 1706
I20260925 22:45:46.259969 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 1344
2026-09-25T22:45:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2434 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:46 dist-test-slave-vdtf krb5kdc[1876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:46 dist-test-slave-vdtf krb5kdc[1876](info): set up 2 sockets
Sep 25 22:45:46 dist-test-slave-vdtf krb5kdc[1876](info): commencing operation
krb5kdc: starting...
W20260925 22:45:47.287561 806 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:45:47 dist-test-slave-vdtf krb5kdc[1876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:45:47.410982 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37137
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:37137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:47.522014 1889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:47.522277 1889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:47.522316 1889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:47.526230 1889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:47.526288 1889 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:45:47.526305 1889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:47.526324 1889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:47.526340 1889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:47.526355 1889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:47.531562 1889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:37137
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:47.532624 1889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:47.533651 1889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:47.542176 1894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:47.542212 1895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:47.542367 1889 server_base.cc:1034] running on GCE node
W20260925 22:45:47.542194 1897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:47.543097 1889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:47.544302 1889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:47.545461 1889 hybrid_clock.cc:648] HybridClock initialized: now 1790376347545450 us; error 37 us; skew 500 ppm
Sep 25 22:45:47 dist-test-slave-vdtf krb5kdc[1876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:47.548307 1889 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:45:47.549209 1889 webserver.cc:460] Webserver started at http://127.0.201.190:43553/ using document root <none> and password file <none>
I20260925 22:45:47.549767 1889 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:47.549860 1889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:47.550096 1889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:47.552204 1889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "8782ae8e93064816ba290722a858a978"
format_stamp: "Formatted at 2026-09-25 22:45:47 on dist-test-slave-vdtf"
server_key: "dd0e6f1aaffbc2a9729c3b944617d288"
server_key_iv: "1e2b31e5c7db22e624a827d0b6f02dcf"
server_key_version: "encryptionkey@0"
I20260925 22:45:47.552778 1889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "8782ae8e93064816ba290722a858a978"
format_stamp: "Formatted at 2026-09-25 22:45:47 on dist-test-slave-vdtf"
server_key: "dd0e6f1aaffbc2a9729c3b944617d288"
server_key_iv: "1e2b31e5c7db22e624a827d0b6f02dcf"
server_key_version: "encryptionkey@0"
I20260925 22:45:47.556509 1889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:45:47.558897 1904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:47.560271 1889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:47.560412 1889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "8782ae8e93064816ba290722a858a978"
format_stamp: "Formatted at 2026-09-25 22:45:47 on dist-test-slave-vdtf"
server_key: "dd0e6f1aaffbc2a9729c3b944617d288"
server_key_iv: "1e2b31e5c7db22e624a827d0b6f02dcf"
server_key_version: "encryptionkey@0"
I20260925 22:45:47.560527 1889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:47.573179 1889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:47.576484 1889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:47.576695 1889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:47.584630 1889 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:37137
I20260925 22:45:47.584648 1966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:37137 every 8 connection(s)
I20260925 22:45:47.585563 1889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:47.587508 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 1889
I20260925 22:45:47.587618 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:45:47.587913 806 external_mini_cluster.cc:1428] Setting key f724453085d1e88358b611be6c3df8a2
I20260925 22:45:47.588594 1967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:47.594424 1967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978: Bootstrap starting.
Sep 25 22:45:47 dist-test-slave-vdtf krb5kdc[1876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:47.596933 1967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:47.597653 1967 log.cc:824] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:47.599735 1967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978: No bootstrap required, opened a new log
I20260925 22:45:47.602670 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:47.589443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.0.201.190:37137)
0925 22:45:47.589892 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:47.589914 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:47.589950 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:45:47.590595 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:47.590612 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:47.590933 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:47.591283 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:47.592088 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:47.593065 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:47.593932 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:47.594399 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:47.594575 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:47.597401 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:47.597406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:47.597419 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:47.597446 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:47.599919 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:47.600372 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:47.600378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:47.600384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:47.600511 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:47.600805 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:47.600808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:47.600810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:47.601190 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:47.601401 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:47.601671 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 22:45:47.601948 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
I20260925 22:45:47.602798 1967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8782ae8e93064816ba290722a858a978" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37137 } }
I20260925 22:45:47.603078 1967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:47.603158 1967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8782ae8e93064816ba290722a858a978, State: Initialized, Role: FOLLOWER
I20260925 22:45:47.603561 1967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8782ae8e93064816ba290722a858a978" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37137 } }
I20260925 22:45:47.603737 1967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:47.603832 1967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:47.603941 1967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:47.605073 1967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8782ae8e93064816ba290722a858a978" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37137 } }
I20260925 22:45:47.605404 1967 leader_election.cc:302] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8782ae8e93064816ba290722a858a978; no voters:
I20260925 22:45:47.605652 1967 leader_election.cc:288] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:47.605803 1972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:47.606027 1972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [term 1 LEADER]: Becoming Leader. State: Replica: 8782ae8e93064816ba290722a858a978, State: Running, Role: LEADER
I20260925 22:45:47.606359 1972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8782ae8e93064816ba290722a858a978" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37137 } }
I20260925 22:45:47.606922 1967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:47.608007 1974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8782ae8e93064816ba290722a858a978. Latest consensus state: current_term: 1 leader_uuid: "8782ae8e93064816ba290722a858a978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8782ae8e93064816ba290722a858a978" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37137 } } }
I20260925 22:45:47.608146 1974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:47.608556 1981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:47.609069 1973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8782ae8e93064816ba290722a858a978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8782ae8e93064816ba290722a858a978" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37137 } } }
I20260925 22:45:47.609180 1973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:47.611184 1981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:47.616837 1981 catalog_manager.cc:1347] Generated new cluster ID: c16ef0170f624ab78792258ddd3db7f7
I20260925 22:45:47.616902 1981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:47.633754 1981 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:47.634892 1981 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:47.646509 1981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978: Generated new TSK 0
I20260925 22:45:47.647207 1981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:45:47.682499 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37137
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:47.793493 1993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:47.793733 1993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:47.793776 1993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:47.797798 1993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:47.797853 1993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:47.797897 1993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:45:47.797961 1993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:45:47.803085 1993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.201.190:37137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:47.804103 1993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:47.805125 1993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:47.814699 1999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:47.814706 1998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:47.814706 2001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:47.814823 1993 server_base.cc:1034] running on GCE node
I20260925 22:45:47.815392 1993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:47.816069 1993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:47.817227 1993 hybrid_clock.cc:648] HybridClock initialized: now 1790376347817210 us; error 27 us; skew 500 ppm
Sep 25 22:45:47 dist-test-slave-vdtf krb5kdc[1876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:47.819968 1993 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:45:47.820781 1993 webserver.cc:460] Webserver started at http://127.0.201.129:46551/ using document root <none> and password file <none>
I20260925 22:45:47.821338 1993 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:47.821430 1993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:47.821635 1993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:47.823590 1993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "b0111aaee4004345a2410f5f659f3095"
format_stamp: "Formatted at 2026-09-25 22:45:47 on dist-test-slave-vdtf"
server_key: "33d0560378c6a300d6fe3ddd75f936c6"
server_key_iv: "1d4400bf4a783d8ad783b50d0d41821f"
server_key_version: "encryptionkey@0"
I20260925 22:45:47.824229 1993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "b0111aaee4004345a2410f5f659f3095"
format_stamp: "Formatted at 2026-09-25 22:45:47 on dist-test-slave-vdtf"
server_key: "33d0560378c6a300d6fe3ddd75f936c6"
server_key_iv: "1d4400bf4a783d8ad783b50d0d41821f"
server_key_version: "encryptionkey@0"
I20260925 22:45:47.827800 1993 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:45:47.830221 2008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:47.831393 1993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:47.831480 1993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "b0111aaee4004345a2410f5f659f3095"
format_stamp: "Formatted at 2026-09-25 22:45:47 on dist-test-slave-vdtf"
server_key: "33d0560378c6a300d6fe3ddd75f936c6"
server_key_iv: "1d4400bf4a783d8ad783b50d0d41821f"
server_key_version: "encryptionkey@0"
I20260925 22:45:47.831614 1993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:47.848188 1993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:47.851058 1993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:47.851256 1993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:47.852771 1993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:47.852818 1993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:47.852859 1993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:47.852909 1993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:47.863785 1993 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:38705
I20260925 22:45:47.863819 2123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:38705 every 8 connection(s)
Sep 25 22:45:47 dist-test-slave-vdtf krb5kdc[1876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:47.864898 1993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:47.868741 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 1993
I20260925 22:45:47.868793 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:47.853555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:39473 (local address 127.0.201.190:37137)
0925 22:45:47.853728 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:47.853735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:47.854547 (+ 812us) server_negotiation.cc:407] Connection header received
0925 22:45:47.856994 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:47.857000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:47.857066 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:47.857131 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:47.858641 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:47.859569 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:47.861476 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:47.861796 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:47.861926 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:47.865073 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:47.865077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:47.865080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:47.865114 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:47.866712 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:47.867254 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:47.867257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:47.867258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:47.867334 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:47.867635 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:47.867637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:47.867638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:47.867879 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:47.867959 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:47.868566 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 22:45:47.868675 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:45:47.868865 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:47.869164 806 external_mini_cluster.cc:1428] Setting key 19fa7c2952ec892afcd417f75fd31cec
I20260925 22:45:47.870042 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:47.853935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37137 (local address 127.0.201.129:39473)
0925 22:45:47.854379 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:47.854430 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:47.856712 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:47.857537 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:47.857544 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:47.857870 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:47.858455 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:47.858480 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:47.859771 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:47.859774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:47.861338 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:47.861350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:47.862006 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:47.862014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:47.862262 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:47.863014 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:47.863033 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:47.864883 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:47.866846 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:47.866850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:47.866862 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:47.867148 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:47.867418 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:47.867420 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:47.867421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:47.867547 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:47.868010 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:47.868015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:47.868283 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 22:45:47.868526 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 22:45:47.868938 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":134,"threads_started":1}
I20260925 22:45:47.871538 2124 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37137
I20260925 22:45:47.871795 2124 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:47.872311 2124 heartbeater.cc:511] Master 127.0.201.190:37137 requested a full tablet report, sending...
I20260925 22:45:47.875094 1921 ts_manager.cc:194] Registered new tserver with Master: b0111aaee4004345a2410f5f659f3095 (127.0.201.129:38705)
I20260925 22:45:47.876616 1921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:39473
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:45:47.883392 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:47.874262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:43025 (local address 127.0.201.190:37137)
0925 22:45:47.874390 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:47.874397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:47.876028 (+ 1631us) server_negotiation.cc:407] Connection header received
0925 22:45:47.876162 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:47.876165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:47.876208 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:47.876293 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:47.877034 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:47.877720 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:47.878587 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:47.878802 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:47.878915 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:47.880134 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:47.880138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:47.880141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:47.880166 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:47.881745 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:47.882176 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:47.882179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:47.882181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:47.882262 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:47.882516 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:47.882518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:47.882519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:47.882752 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:47.882864 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:47.883128 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 22:45:47.883239 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:45:47.883404 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:47.874288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37137 (local address 127.0.201.129:43025)
0925 22:45:47.875901 (+ 1613us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:47.875922 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:47.876053 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:47.876358 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:47.876364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:47.876642 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:47.876821 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:47.876833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:47.877847 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:47.877851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:47.878456 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:47.878464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:47.878921 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:47.878925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:47.879054 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:47.879586 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:47.879597 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:47.879996 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:47.881853 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:47.881855 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:47.881857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:47.882074 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:47.882346 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:47.882348 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:47.882349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:47.882423 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:47.882853 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:47.882856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:47.882951 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 22:45:47.883069 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:45:47.883209 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1508,"mutex_wait_us":26,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:45:47.899489 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37137
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:48.009963 2133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:48.010213 2133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:48.010285 2133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:48.014427 2133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:48.014511 2133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:48.014582 2133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:45:48.014624 2133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:45:48.019629 2133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.201.190:37137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:48.020617 2133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:48.021580 2133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:45:48.031005 2133 server_base.cc:1034] running on GCE node
W20260925 22:45:48.030967 2138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:48.030956 2141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:48.030951 2139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:48.031479 2133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:48.032143 2133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:48.033342 2133 hybrid_clock.cc:648] HybridClock initialized: now 1790376348033324 us; error 29 us; skew 500 ppm
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[1876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:48.036327 2133 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:45:48.037125 2133 webserver.cc:460] Webserver started at http://127.0.201.130:41757/ using document root <none> and password file <none>
I20260925 22:45:48.037711 2133 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:48.037768 2133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:48.037977 2133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:48.040009 2133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "cf6abf67ad4a45fd87ea8a252ac651d8"
format_stamp: "Formatted at 2026-09-25 22:45:48 on dist-test-slave-vdtf"
server_key: "a939ec134dbbf3638241c7e6ac529386"
server_key_iv: "6482604d9cb5731e20e1a349d0baf76d"
server_key_version: "encryptionkey@0"
I20260925 22:45:48.040562 2133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "cf6abf67ad4a45fd87ea8a252ac651d8"
format_stamp: "Formatted at 2026-09-25 22:45:48 on dist-test-slave-vdtf"
server_key: "a939ec134dbbf3638241c7e6ac529386"
server_key_iv: "6482604d9cb5731e20e1a349d0baf76d"
server_key_version: "encryptionkey@0"
I20260925 22:45:48.044036 2133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:45:48.046471 2148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:48.047650 2133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:45:48.047825 2133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "cf6abf67ad4a45fd87ea8a252ac651d8"
format_stamp: "Formatted at 2026-09-25 22:45:48 on dist-test-slave-vdtf"
server_key: "a939ec134dbbf3638241c7e6ac529386"
server_key_iv: "6482604d9cb5731e20e1a349d0baf76d"
server_key_version: "encryptionkey@0"
I20260925 22:45:48.047922 2133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:48.062857 2133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:48.066116 2133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:48.066299 2133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:48.067625 2133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:48.067726 2133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:48.067794 2133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:48.067833 2133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[1876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:48.078480 2133 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:38425
I20260925 22:45:48.079421 2133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:48.080684 2263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:38425 every 8 connection(s)
I20260925 22:45:48.083498 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.068541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:37315 (local address 127.0.201.190:37137)
0925 22:45:48.068666 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.068675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.069692 (+ 1017us) server_negotiation.cc:407] Connection header received
0925 22:45:48.070832 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.070835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.070960 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.071095 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:48.072746 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.073400 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.074564 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.074769 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.074897 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.079799 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:48.079802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:48.079807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:48.079840 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:48.081511 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.082007 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.082011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.082013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.082099 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.082439 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.082444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.082446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.082640 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:48.082752 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.083216 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.083332 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:45:48.084295 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.069076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37137 (local address 127.0.201.130:37315)
0925 22:45:48.069545 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.069585 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.070584 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.071327 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.071339 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.071830 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:48.072574 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.072596 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.073534 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.073537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.074386 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.074430 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.074954 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.074959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.075195 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.076522 (+ 1327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:48.076545 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:48.079580 (+ 3035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:48.081628 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:48.081634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:48.081647 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:48.081887 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:48.082204 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:48.082207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:48.082209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:48.082340 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:48.082748 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:48.082753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:48.082981 (+ 228us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.083162 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.083453 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":131,"threads_started":1}
I20260925 22:45:48.085628 2264 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37137
I20260925 22:45:48.085776 2264 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:48.085760 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 2133
I20260925 22:45:48.085860 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:48.086145 806 external_mini_cluster.cc:1428] Setting key 8313c6396791d949a86bedcc8678b9ac
I20260925 22:45:48.086331 2264 heartbeater.cc:511] Master 127.0.201.190:37137 requested a full tablet report, sending...
I20260925 22:45:48.087345 1921 ts_manager.cc:194] Registered new tserver with Master: cf6abf67ad4a45fd87ea8a252ac651d8 (127.0.201.130:38425)
I20260925 22:45:48.088037 1921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:37315
I20260925 22:45:48.095811 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.088210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:59559 (local address 127.0.201.190:37137)
0925 22:45:48.088314 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.088318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.088643 (+ 325us) server_negotiation.cc:407] Connection header received
0925 22:45:48.088781 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.088784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.088839 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.088922 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:48.089453 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.090211 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.091039 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.091271 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.091388 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.092604 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:48.092607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:48.092610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:48.092632 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:48.094121 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.094544 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.094550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.094552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.094636 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.094910 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.094915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.094917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.095130 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:48.095255 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.095512 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.095619 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:45:48.095825 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.088110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37137 (local address 127.0.201.130:59559)
0925 22:45:48.088537 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.088551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.088681 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.088977 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.088981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.089206 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:48.089323 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.089330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.090346 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.090350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.090910 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.090917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.091389 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.091393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.091482 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.092064 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:48.092075 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:48.092463 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:48.094237 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:48.094244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:48.094246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:48.094437 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:48.094730 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:48.094735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:48.094737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:48.094817 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:48.095244 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:48.095247 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:48.095340 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.095451 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.095612 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:45:48.115666 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37137
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:48.225914 2273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:48.226172 2273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:48.226214 2273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:48.230126 2273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:48.230180 2273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:48.230218 2273 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:45:48.230304 2273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:45:48.235459 2273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.201.190:37137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:48.236527 2273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:48.237556 2273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:48.247208 2279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:48.247246 2273 server_base.cc:1034] running on GCE node
W20260925 22:45:48.247265 2278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:48.247237 2281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:48.247819 2273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:48.248415 2273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:48.249569 2273 hybrid_clock.cc:648] HybridClock initialized: now 1790376348249552 us; error 28 us; skew 500 ppm
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[1876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:48.252377 2273 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:45:48.253247 2273 webserver.cc:460] Webserver started at http://127.0.201.131:42139/ using document root <none> and password file <none>
I20260925 22:45:48.253788 2273 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:48.253846 2273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:48.254058 2273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:48.256098 2273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "e50104e6aa0b485392abac9b8b6bef7c"
format_stamp: "Formatted at 2026-09-25 22:45:48 on dist-test-slave-vdtf"
server_key: "634685f3eb03158dc67c96265c01d143"
server_key_iv: "446c792b0fe25595a5f5b8796d42827c"
server_key_version: "encryptionkey@0"
I20260925 22:45:48.256654 2273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "e50104e6aa0b485392abac9b8b6bef7c"
format_stamp: "Formatted at 2026-09-25 22:45:48 on dist-test-slave-vdtf"
server_key: "634685f3eb03158dc67c96265c01d143"
server_key_iv: "446c792b0fe25595a5f5b8796d42827c"
server_key_version: "encryptionkey@0"
I20260925 22:45:48.260251 2273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:45:48.262604 2288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:48.263911 2273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:45:48.263999 2273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "e50104e6aa0b485392abac9b8b6bef7c"
format_stamp: "Formatted at 2026-09-25 22:45:48 on dist-test-slave-vdtf"
server_key: "634685f3eb03158dc67c96265c01d143"
server_key_iv: "446c792b0fe25595a5f5b8796d42827c"
server_key_version: "encryptionkey@0"
I20260925 22:45:48.264165 2273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:48.284942 2273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:48.288062 2273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:48.288261 2273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:48.289609 2273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:48.289685 2273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:48.289748 2273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:48.289784 2273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[1876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:48.300464 2273 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:46873
I20260925 22:45:48.300858 2403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:46873 every 8 connection(s)
I20260925 22:45:48.301409 2273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:48.301661 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 2273
I20260925 22:45:48.301779 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:48.302098 806 external_mini_cluster.cc:1428] Setting key 496cafd9c1293fa7ec56bc0c762bfb69
I20260925 22:45:48.303982 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.290626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:39185 (local address 127.0.201.190:37137)
0925 22:45:48.290756 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.290760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.291835 (+ 1075us) server_negotiation.cc:407] Connection header received
0925 22:45:48.292906 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.292912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.292979 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.293093 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:48.294586 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.295501 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.296946 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.297251 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.297369 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.300008 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:48.300034 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:48.300040 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:48.300072 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:48.301661 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.302270 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.302274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.302275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.302387 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.302783 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.302787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.302789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.303027 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:48.303124 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.303720 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.303837 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:45:48.304841 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.291169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37137 (local address 127.0.201.131:39185)
0925 22:45:48.291638 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.291722 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.292668 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.293324 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.293335 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.293767 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:48.294426 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.294454 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.295653 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.295658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.296784 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.296793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.297403 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.297407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.297584 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.298169 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:48.298185 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:48.299810 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:48.301778 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:48.301785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:48.301801 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:48.302142 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:48.302504 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:48.302508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:48.302511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:48.302676 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:48.303121 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:48.303126 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:48.303426 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.303610 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.303909 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":142,"threads_started":1}
I20260925 22:45:48.305989 2404 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37137
I20260925 22:45:48.306149 2404 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:48.306703 2404 heartbeater.cc:511] Master 127.0.201.190:37137 requested a full tablet report, sending...
I20260925 22:45:48.307739 1920 ts_manager.cc:194] Registered new tserver with Master: e50104e6aa0b485392abac9b8b6bef7c (127.0.201.131:46873)
I20260925 22:45:48.308475 1920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:39185
I20260925 22:45:48.314504 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.308092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37137 (local address 127.0.201.131:51711)
0925 22:45:48.308345 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.308357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.308465 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.308679 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.308682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.308895 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:48.308991 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.308996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.309723 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.309726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.310266 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.310273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.310715 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.310719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.310803 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.311223 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:48.311233 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:48.311508 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:48.313085 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:48.313090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:48.313092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:48.313278 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:48.313563 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:48.313568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:48.313570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:48.313650 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:48.314015 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:48.314018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:48.314111 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.314212 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.314337 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
I20260925 22:45:48.314510 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.308068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:51711 (local address 127.0.201.190:37137)
0925 22:45:48.308179 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.308182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.308427 (+ 245us) server_negotiation.cc:407] Connection header received
0925 22:45:48.308539 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.308543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.308582 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.308668 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:48.309105 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.309598 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.310384 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.310602 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.310713 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.311637 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:48.311641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:48.311643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:48.311722 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:48.312980 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.313379 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.313384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.313385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.313471 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.313748 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.313751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.313752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.313911 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:48.314014 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.314276 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.314374 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:45:48.316332 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[1876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376348, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[1876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376348, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:48.341288 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.332810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.0.201.190:37137)
0925 22:45:48.332945 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.332948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.332961 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:45:48.333025 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.333027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.333065 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.333166 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:48.333837 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.334420 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.335289 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.335569 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.335723 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.338090 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:48.338094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:48.338096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:48.338122 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:48.339565 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.340045 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.340051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.340053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.340136 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.340424 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.340427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.340429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.340607 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:48.340754 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.341010 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.341120 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:45:48.344303 1920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:45:48.347180 1920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:48.361466 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.355480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:46873 (local address 127.0.0.1:56604)
0925 22:45:48.355849 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.356039 (+ 190us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.356196 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.356836 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.356842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.356866 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:48.357006 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.357012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.358472 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.358478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.359873 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.359883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.360940 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.360944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.361048 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.361088 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.361201 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.361315 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":126,"threads_started":1}
I20260925 22:45:48.361479 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.355596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.0.201.131:46873)
0925 22:45:48.355960 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.355970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.356145 (+ 175us) server_negotiation.cc:407] Connection header received
0925 22:45:48.356404 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.356412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.356644 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.356819 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:48.357129 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.358360 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.360001 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.360808 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.360912 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.361064 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.361246 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.361336 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":156,"threads_started":1}
I20260925 22:45:48.361991 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.356646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:38705 (local address 127.0.0.1:50392)
0925 22:45:48.357695 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.357706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.357781 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.358173 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.358175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.358187 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:48.358244 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.358248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.359549 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.359553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.360459 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.360467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.361607 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.361610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.361698 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.361725 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.361797 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.361866 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260925 22:45:48.362401 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.356297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:38425 (local address 127.0.0.1:60010)
0925 22:45:48.357430 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.357444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.357536 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.357940 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.357943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.357959 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:48.358053 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.358058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.359398 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.359402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.360533 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.360544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.361950 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.361953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.362130 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.362152 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.362219 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.362278 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":99,"threads_started":1}
I20260925 22:45:48.362628 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.357044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.0.201.129:38705)
0925 22:45:48.357348 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.357360 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.357795 (+ 435us) server_negotiation.cc:407] Connection header received
0925 22:45:48.357874 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.357881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.358055 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.358202 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:48.358326 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.359433 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.360590 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.361293 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.361386 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.361542 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.361828 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.362476 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":98,"threads_started":1}
I20260925 22:45:48.362926 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.356467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.0.201.130:38425)
0925 22:45:48.356886 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.356896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.357539 (+ 643us) server_negotiation.cc:407] Connection header received
0925 22:45:48.357627 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.357631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.357802 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.357944 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:48.358146 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.359274 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.360671 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.361460 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.361747 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.361900 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.362714 (+ 814us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.362798 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":104,"threads_started":1}
I20260925 22:45:48.364584 2338 tablet_service.cc:1467] Processing CreateTablet for tablet 93e8b14d3c7f4121a49b1c06ee42ec2f (DEFAULT_TABLE table=test-table [id=33d246399b214cf68bc50440285b0a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:48.365630 2338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e8b14d3c7f4121a49b1c06ee42ec2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:48.365339 2198 tablet_service.cc:1467] Processing CreateTablet for tablet 93e8b14d3c7f4121a49b1c06ee42ec2f (DEFAULT_TABLE table=test-table [id=33d246399b214cf68bc50440285b0a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:48.365337 2042 tablet_service.cc:1467] Processing CreateTablet for tablet 93e8b14d3c7f4121a49b1c06ee42ec2f (DEFAULT_TABLE table=test-table [id=33d246399b214cf68bc50440285b0a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:48.366298 2198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e8b14d3c7f4121a49b1c06ee42ec2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:48.366324 2042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e8b14d3c7f4121a49b1c06ee42ec2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:48.372984 2427 tablet_bootstrap.cc:492] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: Bootstrap starting.
I20260925 22:45:48.374301 2428 tablet_bootstrap.cc:492] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: Bootstrap starting.
I20260925 22:45:48.375559 2427 tablet_bootstrap.cc:654] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:48.376329 2428 tablet_bootstrap.cc:654] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:48.376448 2427 log.cc:824] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:48.377063 2428 log.cc:824] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:48.377687 2429 tablet_bootstrap.cc:492] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: Bootstrap starting.
I20260925 22:45:48.378405 2427 tablet_bootstrap.cc:492] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: No bootstrap required, opened a new log
I20260925 22:45:48.378592 2427 ts_tablet_manager.cc:1397] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:45:48.378986 2428 tablet_bootstrap.cc:492] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: No bootstrap required, opened a new log
I20260925 22:45:48.379156 2428 ts_tablet_manager.cc:1397] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:45:48.381175 2429 tablet_bootstrap.cc:654] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:48.381879 2429 log.cc:824] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:48.382491 2427 raft_consensus.cc:348] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } }
I20260925 22:45:48.382746 2427 raft_consensus.cc:374] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:48.382802 2427 raft_consensus.cc:729] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50104e6aa0b485392abac9b8b6bef7c, State: Initialized, Role: FOLLOWER
I20260925 22:45:48.382863 2428 raft_consensus.cc:348] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } }
I20260925 22:45:48.383080 2428 raft_consensus.cc:374] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:48.383131 2428 raft_consensus.cc:729] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6abf67ad4a45fd87ea8a252ac651d8, State: Initialized, Role: FOLLOWER
I20260925 22:45:48.383235 2427 consensus_queue.cc:260] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } }
I20260925 22:45:48.383522 2428 consensus_queue.cc:260] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } }
I20260925 22:45:48.383744 2429 tablet_bootstrap.cc:492] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: No bootstrap required, opened a new log
I20260925 22:45:48.383924 2429 ts_tablet_manager.cc:1397] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:45:48.383986 2404 heartbeater.cc:503] Master 127.0.201.190:37137 was elected leader, sending a full tablet report...
I20260925 22:45:48.384353 2427 ts_tablet_manager.cc:1428] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:45:48.385426 2428 ts_tablet_manager.cc:1428] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:45:48.385733 2264 heartbeater.cc:503] Master 127.0.201.190:37137 was elected leader, sending a full tablet report...
I20260925 22:45:48.387666 2429 raft_consensus.cc:348] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } }
I20260925 22:45:48.387964 2429 raft_consensus.cc:374] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:48.388024 2429 raft_consensus.cc:729] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0111aaee4004345a2410f5f659f3095, State: Initialized, Role: FOLLOWER
I20260925 22:45:48.388460 2429 consensus_queue.cc:260] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } }
I20260925 22:45:48.389009 2124 heartbeater.cc:503] Master 127.0.201.190:37137 was elected leader, sending a full tablet report...
I20260925 22:45:48.389298 2429 ts_tablet_manager.cc:1428] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:45:48.432116 2433 raft_consensus.cc:482] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:48.432331 2433 raft_consensus.cc:504] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } }
I20260925 22:45:48.433519 2433 leader_election.cc:288] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf6abf67ad4a45fd87ea8a252ac651d8 (127.0.201.130:38425), b0111aaee4004345a2410f5f659f3095 (127.0.201.129:38705)
I20260925 22:45:48.437414 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.433898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:54833 (local address 127.0.201.130:38425)
0925 22:45:48.434058 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.434064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.434083 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:45:48.434139 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.434142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.434199 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.434282 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:48.434662 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.435236 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.436456 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.436842 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.436985 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.437045 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.437229 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.437294 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:45:48.437415 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.433785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:38425 (local address 127.0.201.131:54833)
0925 22:45:48.433927 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.433944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.434038 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.434403 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.434406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.434423 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:48.434562 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.434572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.435350 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.435356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.436328 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.436336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.436972 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.436976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.437088 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.437109 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.437206 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.437282 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 22:45:48.438030 2218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e8b14d3c7f4121a49b1c06ee42ec2f" candidate_uuid: "e50104e6aa0b485392abac9b8b6bef7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" is_pre_election: true
I20260925 22:45:48.438103 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.434206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:44527 (local address 127.0.201.129:38705)
0925 22:45:48.434335 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.434340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.434530 (+ 190us) server_negotiation.cc:407] Connection header received
0925 22:45:48.434656 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.434659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.434703 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.434801 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:48.435084 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.435757 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.437134 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.437628 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.437730 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.437771 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.437957 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.438012 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:45:48.438123 2436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.434083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:38705 (local address 127.0.201.131:44527)
0925 22:45:48.434441 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.434452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.434569 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.434828 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.434831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.434845 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:48.434968 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.434973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.435870 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.435877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.437004 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.437013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.437735 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.437740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.437835 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.437856 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.437932 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.437990 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":149,"threads_started":1}
I20260925 22:45:48.438338 2218 raft_consensus.cc:2454] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e50104e6aa0b485392abac9b8b6bef7c in term 0.
I20260925 22:45:48.438748 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e8b14d3c7f4121a49b1c06ee42ec2f" candidate_uuid: "e50104e6aa0b485392abac9b8b6bef7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0111aaee4004345a2410f5f659f3095" is_pre_election: true
I20260925 22:45:48.438814 2289 leader_election.cc:302] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6abf67ad4a45fd87ea8a252ac651d8, e50104e6aa0b485392abac9b8b6bef7c; no voters:
I20260925 22:45:48.439030 2078 raft_consensus.cc:2454] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e50104e6aa0b485392abac9b8b6bef7c in term 0.
I20260925 22:45:48.439067 2433 raft_consensus.cc:2781] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:48.439181 2433 raft_consensus.cc:482] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:48.439262 2433 raft_consensus.cc:3037] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:48.440369 2433 raft_consensus.cc:504] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } }
I20260925 22:45:48.440709 2433 leader_election.cc:288] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [CANDIDATE]: Term 1 election: Requested vote from peers cf6abf67ad4a45fd87ea8a252ac651d8 (127.0.201.130:38425), b0111aaee4004345a2410f5f659f3095 (127.0.201.129:38705)
I20260925 22:45:48.441115 2218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e8b14d3c7f4121a49b1c06ee42ec2f" candidate_uuid: "e50104e6aa0b485392abac9b8b6bef7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8"
I20260925 22:45:48.441179 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e8b14d3c7f4121a49b1c06ee42ec2f" candidate_uuid: "e50104e6aa0b485392abac9b8b6bef7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0111aaee4004345a2410f5f659f3095"
I20260925 22:45:48.441237 2218 raft_consensus.cc:3037] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:48.441355 2078 raft_consensus.cc:3037] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:48.442358 2078 raft_consensus.cc:2454] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e50104e6aa0b485392abac9b8b6bef7c in term 1.
I20260925 22:45:48.442654 2289 leader_election.cc:302] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0111aaee4004345a2410f5f659f3095, e50104e6aa0b485392abac9b8b6bef7c; no voters:
I20260925 22:45:48.442842 2218 raft_consensus.cc:2454] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e50104e6aa0b485392abac9b8b6bef7c in term 1.
I20260925 22:45:48.442838 2433 raft_consensus.cc:2781] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:48.443080 2433 raft_consensus.cc:686] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 1 LEADER]: Becoming Leader. State: Replica: e50104e6aa0b485392abac9b8b6bef7c, State: Running, Role: LEADER
I20260925 22:45:48.443420 2433 consensus_queue.cc:237] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } }
I20260925 22:45:48.446626 1921 catalog_manager.cc:5495] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c reported cstate change: term changed from 0 to 1, leader changed from <none> to e50104e6aa0b485392abac9b8b6bef7c (127.0.201.131). New cstate: current_term: 1 leader_uuid: "e50104e6aa0b485392abac9b8b6bef7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } health_report { overall_health: UNKNOWN } } }
I20260925 22:45:48.464396 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.460630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.0.201.131:46873)
0925 22:45:48.460772 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.460779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.460794 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:45:48.461022 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.461026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.461080 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.461186 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:48.461443 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.461957 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.462868 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.463087 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.463201 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.463518 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:48.463982 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:48.464100 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.464214 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.464265 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:45:48.469880 2078 raft_consensus.cc:1261] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095 [term 1 FOLLOWER]: Refusing update from remote peer e50104e6aa0b485392abac9b8b6bef7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:48.469880 2218 raft_consensus.cc:1261] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 FOLLOWER]: Refusing update from remote peer e50104e6aa0b485392abac9b8b6bef7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:48.470578 2437 consensus_queue.cc:1035] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:48.470743 2433 consensus_queue.cc:1035] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:48.478222 2441 mvcc.cc:205] Tried to move back new op lower bound from 7333381523325636608 to 7333381523227611136. Current Snapshot: MvccSnapshot[applied={T|T < 7333381523325636608}]
I20260925 22:45:48.490211 1920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:48.490389 1920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:48.492975 1920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978: Sending DeleteTablet for 3 replicas of tablet 93e8b14d3c7f4121a49b1c06ee42ec2f
I20260925 22:45:48.493717 2338 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e8b14d3c7f4121a49b1c06ee42ec2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:48 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:56604
I20260925 22:45:48.493846 2198 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e8b14d3c7f4121a49b1c06ee42ec2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:48 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:60010
I20260925 22:45:48.493974 2042 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e8b14d3c7f4121a49b1c06ee42ec2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:48 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:50392
I20260925 22:45:48.494179 2450 tablet_replica.cc:330] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: stopping tablet replica
I20260925 22:45:48.494395 2451 tablet_replica.cc:330] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: stopping tablet replica
I20260925 22:45:48.494474 2450 raft_consensus.cc:2229] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:48.494738 2451 raft_consensus.cc:2229] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:48.494870 2450 raft_consensus.cc:2258] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:48.495117 2451 raft_consensus.cc:2258] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:48.495071 1920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:45:48.495241 2452 tablet_replica.cc:330] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: stopping tablet replica
W20260925 22:45:48.495477 1920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:48.495525 2452 raft_consensus.cc:2229] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:48.495827 2452 raft_consensus.cc:2258] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:48.497298 2452 ts_tablet_manager.cc:1905] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:48.497653 2451 ts_tablet_manager.cc:1905] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:48.500095 2452 ts_tablet_manager.cc:1918] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:48.500186 2452 log.cc:1196] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/93e8b14d3c7f4121a49b1c06ee42ec2f
I20260925 22:45:48.500590 2452 ts_tablet_manager.cc:1939] T 93e8b14d3c7f4121a49b1c06ee42ec2f P b0111aaee4004345a2410f5f659f3095: Deleting consensus metadata
I20260925 22:45:48.500667 2042 tablet_service.cc:1467] Processing CreateTablet for tablet 4f56af0413f84cd4aa986f47f716fcd3 (DEFAULT_TABLE table=test-table [id=6040a05b960340eca25579b91dea883a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:48.500923 2042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f56af0413f84cd4aa986f47f716fcd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:48.501660 2450 ts_tablet_manager.cc:1905] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:48.502935 2429 tablet_bootstrap.cc:492] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: Bootstrap starting.
I20260925 22:45:48.503124 2198 tablet_service.cc:1467] Processing CreateTablet for tablet 4f56af0413f84cd4aa986f47f716fcd3 (DEFAULT_TABLE table=test-table [id=6040a05b960340eca25579b91dea883a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:48.503346 2198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f56af0413f84cd4aa986f47f716fcd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:48.503906 2429 tablet_bootstrap.cc:654] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:48.503978 2450 ts_tablet_manager.cc:1918] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:48.504060 2450 log.cc:1196] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/93e8b14d3c7f4121a49b1c06ee42ec2f
I20260925 22:45:48.504356 2450 ts_tablet_manager.cc:1939] T 93e8b14d3c7f4121a49b1c06ee42ec2f P e50104e6aa0b485392abac9b8b6bef7c: Deleting consensus metadata
I20260925 22:45:48.504331 2338 tablet_service.cc:1467] Processing CreateTablet for tablet 4f56af0413f84cd4aa986f47f716fcd3 (DEFAULT_TABLE table=test-table [id=6040a05b960340eca25579b91dea883a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:48.504583 2338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f56af0413f84cd4aa986f47f716fcd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:48.506214 2428 tablet_bootstrap.cc:492] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: Bootstrap starting.
I20260925 22:45:48.506647 2427 tablet_bootstrap.cc:492] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: Bootstrap starting.
I20260925 22:45:48.506882 2451 ts_tablet_manager.cc:1918] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:48.506961 2451 log.cc:1196] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/93e8b14d3c7f4121a49b1c06ee42ec2f
I20260925 22:45:48.507210 2428 tablet_bootstrap.cc:654] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:48.507259 2451 ts_tablet_manager.cc:1939] T 93e8b14d3c7f4121a49b1c06ee42ec2f P cf6abf67ad4a45fd87ea8a252ac651d8: Deleting consensus metadata
I20260925 22:45:48.507587 2427 tablet_bootstrap.cc:654] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:48.508533 1905 catalog_manager.cc:4841] TS cf6abf67ad4a45fd87ea8a252ac651d8 (127.0.201.130:38425): tablet 93e8b14d3c7f4121a49b1c06ee42ec2f (table test-table [id=33d246399b214cf68bc50440285b0a4f]) successfully deleted
I20260925 22:45:48.509128 1905 catalog_manager.cc:4841] TS e50104e6aa0b485392abac9b8b6bef7c (127.0.201.131:46873): tablet 93e8b14d3c7f4121a49b1c06ee42ec2f (table test-table [id=33d246399b214cf68bc50440285b0a4f]) successfully deleted
I20260925 22:45:48.509529 2428 tablet_bootstrap.cc:492] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: No bootstrap required, opened a new log
I20260925 22:45:48.509604 2428 ts_tablet_manager.cc:1397] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 22:45:48.510000 1905 catalog_manager.cc:4841] TS b0111aaee4004345a2410f5f659f3095 (127.0.201.129:38705): tablet 93e8b14d3c7f4121a49b1c06ee42ec2f (table test-table [id=33d246399b214cf68bc50440285b0a4f]) successfully deleted
I20260925 22:45:48.510043 2428 raft_consensus.cc:348] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.510160 2428 raft_consensus.cc:374] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:48.510195 2428 raft_consensus.cc:729] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6abf67ad4a45fd87ea8a252ac651d8, State: Initialized, Role: FOLLOWER
I20260925 22:45:48.510314 2428 consensus_queue.cc:260] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.510532 2427 tablet_bootstrap.cc:492] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: No bootstrap required, opened a new log
I20260925 22:45:48.510600 2427 ts_tablet_manager.cc:1397] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:45:48.510603 2428 ts_tablet_manager.cc:1428] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:45:48.510814 2429 tablet_bootstrap.cc:492] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: No bootstrap required, opened a new log
I20260925 22:45:48.510891 2429 ts_tablet_manager.cc:1397] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260925 22:45:48.510984 2427 raft_consensus.cc:348] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.511101 2427 raft_consensus.cc:374] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:48.511132 2427 raft_consensus.cc:729] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50104e6aa0b485392abac9b8b6bef7c, State: Initialized, Role: FOLLOWER
I20260925 22:45:48.511235 2427 consensus_queue.cc:260] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.511464 2427 ts_tablet_manager.cc:1428] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:45:48.511654 2429 raft_consensus.cc:348] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.511809 2429 raft_consensus.cc:374] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:48.511843 2429 raft_consensus.cc:729] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0111aaee4004345a2410f5f659f3095, State: Initialized, Role: FOLLOWER
I20260925 22:45:48.511945 2429 consensus_queue.cc:260] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.512588 2429 ts_tablet_manager.cc:1428] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260925 22:45:48.552413 2405 tablet.cc:2369] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:45:48.581293 2265 tablet.cc:2369] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:45:48.616698 2125 tablet.cc:2369] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:45:48.617163 2446 raft_consensus.cc:482] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:48.617292 2446 raft_consensus.cc:504] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.617794 2446 leader_election.cc:288] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0111aaee4004345a2410f5f659f3095 (127.0.201.129:38705), cf6abf67ad4a45fd87ea8a252ac651d8 (127.0.201.130:38425)
I20260925 22:45:48.618328 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f56af0413f84cd4aa986f47f716fcd3" candidate_uuid: "e50104e6aa0b485392abac9b8b6bef7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0111aaee4004345a2410f5f659f3095" is_pre_election: true
I20260925 22:45:48.618353 2218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f56af0413f84cd4aa986f47f716fcd3" candidate_uuid: "e50104e6aa0b485392abac9b8b6bef7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" is_pre_election: true
I20260925 22:45:48.618507 2078 raft_consensus.cc:2454] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e50104e6aa0b485392abac9b8b6bef7c in term 0.
I20260925 22:45:48.618513 2218 raft_consensus.cc:2454] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e50104e6aa0b485392abac9b8b6bef7c in term 0.
I20260925 22:45:48.618798 2289 leader_election.cc:302] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0111aaee4004345a2410f5f659f3095, e50104e6aa0b485392abac9b8b6bef7c; no voters:
I20260925 22:45:48.619004 2446 raft_consensus.cc:2781] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:48.619086 2446 raft_consensus.cc:482] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:48.619138 2446 raft_consensus.cc:3037] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:48.620267 2446 raft_consensus.cc:504] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.620616 2446 leader_election.cc:288] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [CANDIDATE]: Term 1 election: Requested vote from peers b0111aaee4004345a2410f5f659f3095 (127.0.201.129:38705), cf6abf67ad4a45fd87ea8a252ac651d8 (127.0.201.130:38425)
I20260925 22:45:48.621040 2218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f56af0413f84cd4aa986f47f716fcd3" candidate_uuid: "e50104e6aa0b485392abac9b8b6bef7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8"
I20260925 22:45:48.621049 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f56af0413f84cd4aa986f47f716fcd3" candidate_uuid: "e50104e6aa0b485392abac9b8b6bef7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0111aaee4004345a2410f5f659f3095"
I20260925 22:45:48.621171 2078 raft_consensus.cc:3037] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:48.621177 2218 raft_consensus.cc:3037] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:48.622155 2218 raft_consensus.cc:2454] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e50104e6aa0b485392abac9b8b6bef7c in term 1.
I20260925 22:45:48.622387 2078 raft_consensus.cc:2454] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e50104e6aa0b485392abac9b8b6bef7c in term 1.
I20260925 22:45:48.622515 2289 leader_election.cc:302] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6abf67ad4a45fd87ea8a252ac651d8, e50104e6aa0b485392abac9b8b6bef7c; no voters:
I20260925 22:45:48.622668 2446 raft_consensus.cc:2781] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:48.622762 2446 raft_consensus.cc:686] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 1 LEADER]: Becoming Leader. State: Replica: e50104e6aa0b485392abac9b8b6bef7c, State: Running, Role: LEADER
I20260925 22:45:48.622903 2446 consensus_queue.cc:237] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.624917 1917 catalog_manager.cc:5495] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c reported cstate change: term changed from 0 to 1, leader changed from <none> to e50104e6aa0b485392abac9b8b6bef7c (127.0.201.131). New cstate: current_term: 1 leader_uuid: "e50104e6aa0b485392abac9b8b6bef7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } health_report { overall_health: HEALTHY } } }
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[1876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:48.662030 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.653773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.0.201.190:37137)
0925 22:45:48.653917 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.653922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.654177 (+ 255us) server_negotiation.cc:407] Connection header received
0925 22:45:48.654260 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.654263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.654310 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.654406 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:48.655119 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.655913 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.656679 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.656881 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.656958 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.659231 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:48.659234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:48.659236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:48.659256 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:48.660569 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.660974 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.660976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.660977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.661045 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.661291 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.661293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.661294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.661450 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:48.661522 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.661783 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.661895 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":50}
I20260925 22:45:48.662036 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.653701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37137 (local address 127.0.0.1:60732)
0925 22:45:48.654071 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.654083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.654186 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.654463 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.654465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.654862 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:48.654959 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.654964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.656028 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.656030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.656569 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.656576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.656982 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.656984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.657057 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.657706 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:48.657718 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:48.659104 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:48.660664 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:48.660671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:48.660676 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:48.660890 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:48.661135 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:48.661137 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:48.661139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:48.661211 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:48.661537 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:48.661542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:48.661639 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.661728 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.661842 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260925 22:45:48.665997 1917 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:60732:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:45:48.666751 1917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:48.671530 2042 tablet_service.cc:1467] Processing CreateTablet for tablet 06d253b870d944cea88e14ff0714f880 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a77844cde5a44ffdaeca2b835bbbd64b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:45:48.671515 2198 tablet_service.cc:1467] Processing CreateTablet for tablet 06d253b870d944cea88e14ff0714f880 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a77844cde5a44ffdaeca2b835bbbd64b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:45:48.671528 2338 tablet_service.cc:1467] Processing CreateTablet for tablet 06d253b870d944cea88e14ff0714f880 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a77844cde5a44ffdaeca2b835bbbd64b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:45:48.671912 2338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d253b870d944cea88e14ff0714f880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:48.671918 2198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d253b870d944cea88e14ff0714f880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:48.671918 2042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d253b870d944cea88e14ff0714f880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:48.674204 2427 tablet_bootstrap.cc:492] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: Bootstrap starting.
I20260925 22:45:48.674578 2428 tablet_bootstrap.cc:492] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: Bootstrap starting.
I20260925 22:45:48.674588 2429 tablet_bootstrap.cc:492] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: Bootstrap starting.
I20260925 22:45:48.675117 2427 tablet_bootstrap.cc:654] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:48.675472 2428 tablet_bootstrap.cc:654] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:48.675472 2429 tablet_bootstrap.cc:654] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:48.676613 2427 tablet_bootstrap.cc:492] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: No bootstrap required, opened a new log
I20260925 22:45:48.676697 2427 ts_tablet_manager.cc:1397] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:45:48.676764 2429 tablet_bootstrap.cc:492] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: No bootstrap required, opened a new log
I20260925 22:45:48.676831 2429 ts_tablet_manager.cc:1397] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 22:45:48.677008 2428 tablet_bootstrap.cc:492] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: No bootstrap required, opened a new log
I20260925 22:45:48.677088 2428 ts_tablet_manager.cc:1397] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 22:45:48.677129 2427 raft_consensus.cc:348] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.677242 2427 raft_consensus.cc:374] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:48.677207 2429 raft_consensus.cc:348] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.677274 2427 raft_consensus.cc:729] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50104e6aa0b485392abac9b8b6bef7c, State: Initialized, Role: FOLLOWER
I20260925 22:45:48.677301 2429 raft_consensus.cc:374] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:48.677335 2429 raft_consensus.cc:729] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0111aaee4004345a2410f5f659f3095, State: Initialized, Role: FOLLOWER
I20260925 22:45:48.677403 2427 consensus_queue.cc:260] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.677441 2429 consensus_queue.cc:260] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.677428 2428 raft_consensus.cc:348] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.677520 2428 raft_consensus.cc:374] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:48.677575 2428 raft_consensus.cc:729] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6abf67ad4a45fd87ea8a252ac651d8, State: Initialized, Role: FOLLOWER
I20260925 22:45:48.677676 2429 ts_tablet_manager.cc:1428] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:45:48.677687 2428 consensus_queue.cc:260] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.677670 2437 tablet_replica.cc:439] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } }
I20260925 22:45:48.677799 2437 tablet_replica.cc:442] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:48.677783 2435 tablet_replica.cc:439] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } }
I20260925 22:45:48.677903 2435 tablet_replica.cc:442] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:48.677925 2428 ts_tablet_manager.cc:1428] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:45:48.678444 2434 tablet_replica.cc:439] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } }
I20260925 22:45:48.678563 2434 tablet_replica.cc:442] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:48.677677 2427 ts_tablet_manager.cc:1428] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:45:48.714851 2434 raft_consensus.cc:482] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:48.715034 2434 raft_consensus.cc:504] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.716189 2434 leader_election.cc:288] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0111aaee4004345a2410f5f659f3095 (127.0.201.129:38705), e50104e6aa0b485392abac9b8b6bef7c (127.0.201.131:46873)
I20260925 22:45:48.721421 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.716633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:37657 (local address 127.0.201.129:38705)
0925 22:45:48.716795 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.716799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.716907 (+ 108us) server_negotiation.cc:407] Connection header received
0925 22:45:48.717070 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.717073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.717128 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.717215 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:48.717555 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.718349 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.720134 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.720720 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.720839 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.720888 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.721229 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.721297 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:45:48.721421 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.717065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:53343 (local address 127.0.201.131:46873)
0925 22:45:48.717207 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.717212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.717279 (+ 67us) server_negotiation.cc:407] Connection header received
0925 22:45:48.717422 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.717424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.717463 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.717564 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:48.717795 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.718487 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.720134 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.720719 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.720839 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.720888 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.721229 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.721297 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:45:48.721521 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.716499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:38705 (local address 127.0.201.130:37657)
0925 22:45:48.716803 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.716820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.716956 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.717252 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.717258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.717295 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:48.717420 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.717426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.718458 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.718460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.720004 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.720017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.720843 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.720846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.721060 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.721116 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.721194 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.721329 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":240,"thread_start_us":105,"threads_started":1}
I20260925 22:45:48.721524 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.716918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:46873 (local address 127.0.201.130:53343)
0925 22:45:48.717214 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.717227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.717311 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.717605 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.717607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.717621 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:48.717691 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.717695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.718581 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.718585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.720002 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.720013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.720843 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.720846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.721056 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.721116 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.721194 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.721332 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":195,"spinlock_wait_cycles":896,"thread_start_us":78,"threads_started":1}
I20260925 22:45:48.721995 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d253b870d944cea88e14ff0714f880" candidate_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0111aaee4004345a2410f5f659f3095" is_pre_election: true
I20260925 22:45:48.722086 2358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d253b870d944cea88e14ff0714f880" candidate_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50104e6aa0b485392abac9b8b6bef7c" is_pre_election: true
I20260925 22:45:48.722154 2078 raft_consensus.cc:2454] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6abf67ad4a45fd87ea8a252ac651d8 in term 0.
I20260925 22:45:48.722353 2358 raft_consensus.cc:2454] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6abf67ad4a45fd87ea8a252ac651d8 in term 0.
I20260925 22:45:48.722551 2149 leader_election.cc:302] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0111aaee4004345a2410f5f659f3095, cf6abf67ad4a45fd87ea8a252ac651d8; no voters:
I20260925 22:45:48.722815 2434 raft_consensus.cc:2781] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:48.722893 2434 raft_consensus.cc:482] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:48.722931 2434 raft_consensus.cc:3037] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:48.724009 2434 raft_consensus.cc:504] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.724323 2434 leader_election.cc:288] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [CANDIDATE]: Term 1 election: Requested vote from peers b0111aaee4004345a2410f5f659f3095 (127.0.201.129:38705), e50104e6aa0b485392abac9b8b6bef7c (127.0.201.131:46873)
I20260925 22:45:48.724624 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d253b870d944cea88e14ff0714f880" candidate_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0111aaee4004345a2410f5f659f3095"
I20260925 22:45:48.724663 2358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d253b870d944cea88e14ff0714f880" candidate_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50104e6aa0b485392abac9b8b6bef7c"
I20260925 22:45:48.724730 2078 raft_consensus.cc:3037] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:48.724819 2358 raft_consensus.cc:3037] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:48.725808 2078 raft_consensus.cc:2454] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6abf67ad4a45fd87ea8a252ac651d8 in term 1.
I20260925 22:45:48.725946 2358 raft_consensus.cc:2454] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6abf67ad4a45fd87ea8a252ac651d8 in term 1.
I20260925 22:45:48.726152 2149 leader_election.cc:302] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0111aaee4004345a2410f5f659f3095, cf6abf67ad4a45fd87ea8a252ac651d8; no voters:
I20260925 22:45:48.726315 2434 raft_consensus.cc:2781] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:48.726485 2434 raft_consensus.cc:686] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 LEADER]: Becoming Leader. State: Replica: cf6abf67ad4a45fd87ea8a252ac651d8, State: Running, Role: LEADER
I20260925 22:45:48.726740 2434 consensus_queue.cc:237] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } }
I20260925 22:45:48.727833 2468 tablet_replica.cc:439] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: TxnStatusTablet state changed. Reason: New leader cf6abf67ad4a45fd87ea8a252ac651d8. Latest consensus state: current_term: 1 leader_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } }
I20260925 22:45:48.727957 2468 tablet_replica.cc:442] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:45:48.728519 2470 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:45:48.728681 2470 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:45:48.728996 1917 catalog_manager.cc:5495] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf6abf67ad4a45fd87ea8a252ac651d8 (127.0.201.130). New cstate: current_term: 1 leader_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } health_report { overall_health: UNKNOWN } } }
I20260925 22:45:48.747042 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.743243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.0.201.130:38425)
0925 22:45:48.743364 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.743368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.743429 (+ 61us) server_negotiation.cc:407] Connection header received
0925 22:45:48.743634 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.743636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.743739 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.743828 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:48.744090 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.744576 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.745750 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.745947 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.746020 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.746321 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:48.746695 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:48.746777 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.746896 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.746941 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:45:48.747087 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.743150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:38425 (local address 127.0.0.1:60016)
0925 22:45:48.743371 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.743385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.743492 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.743866 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.743869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.743878 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:45:48.743983 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.743989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.744679 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.744682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.745641 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.745651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.746048 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.746052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.746143 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.746193 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:45:48.746790 (+ 597us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:45:48.746804 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.746876 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.746953 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260925 22:45:48.749863 2078 raft_consensus.cc:1261] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095 [term 1 FOLLOWER]: Refusing update from remote peer cf6abf67ad4a45fd87ea8a252ac651d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:48.750094 2358 raft_consensus.cc:1261] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c [term 1 FOLLOWER]: Refusing update from remote peer cf6abf67ad4a45fd87ea8a252ac651d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:48.751593 2434 consensus_queue.cc:1035] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:48.751783 2468 consensus_queue.cc:1035] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:48.753269 2435 tablet_replica.cc:439] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: TxnStatusTablet state changed. Reason: New leader cf6abf67ad4a45fd87ea8a252ac651d8. Latest consensus state: current_term: 1 leader_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } }
I20260925 22:45:48.753408 2435 tablet_replica.cc:442] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:48.753916 2437 tablet_replica.cc:439] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: TxnStatusTablet state changed. Reason: New leader cf6abf67ad4a45fd87ea8a252ac651d8. Latest consensus state: current_term: 1 leader_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } }
I20260925 22:45:48.754031 2437 tablet_replica.cc:442] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:48.755231 2468 tablet_replica.cc:439] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } }
I20260925 22:45:48.755334 2468 tablet_replica.cc:442] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:45:48.755615 2435 tablet_replica.cc:439] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } }
I20260925 22:45:48.755731 2435 tablet_replica.cc:442] T 06d253b870d944cea88e14ff0714f880 P b0111aaee4004345a2410f5f659f3095: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:48.758347 2442 mvcc.cc:205] Tried to move back new op lower bound from 7333381524474490880 to 7333381524386865152. Current Snapshot: MvccSnapshot[applied={T|T < 7333381524474490880 or (T in {7333381524474490880})}]
I20260925 22:45:48.759913 2468 tablet_replica.cc:439] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } }
I20260925 22:45:48.760104 2468 tablet_replica.cc:442] T 06d253b870d944cea88e14ff0714f880 P cf6abf67ad4a45fd87ea8a252ac651d8: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:45:48.762128 2437 tablet_replica.cc:439] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 } } peers { permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 } } peers { permanent_uuid: "e50104e6aa0b485392abac9b8b6bef7c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46873 } } }
I20260925 22:45:48.762240 2437 tablet_replica.cc:442] T 06d253b870d944cea88e14ff0714f880 P e50104e6aa0b485392abac9b8b6bef7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:48.770547 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.767323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:43965 (local address 127.0.201.130:38425)
0925 22:45:48.767485 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.767490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.767505 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:45:48.767631 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.767633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.767669 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.767776 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:48.768027 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.768657 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.769521 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.769749 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.769861 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.770129 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:48.770203 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:48.770280 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.770370 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.770413 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:45:48.770548 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.767279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:38425 (local address 127.0.201.131:43965)
0925 22:45:48.767411 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.767426 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.767535 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.767787 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.767790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.767798 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:45:48.767896 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.767901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.768766 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.768769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.769390 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.769397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.769856 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.769860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.769943 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.769991 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:45:48.770294 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:45:48.770301 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.770360 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.770442 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 22:45:48.777338 2218 raft_consensus.cc:1261] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 FOLLOWER]: Refusing update from remote peer e50104e6aa0b485392abac9b8b6bef7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:48.777384 2078 raft_consensus.cc:1261] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095 [term 1 FOLLOWER]: Refusing update from remote peer e50104e6aa0b485392abac9b8b6bef7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:48.777753 2446 consensus_queue.cc:1035] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6abf67ad4a45fd87ea8a252ac651d8" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:48.777931 2437 consensus_queue.cc:1035] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0111aaee4004345a2410f5f659f3095" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:48.784483 2440 mvcc.cc:205] Tried to move back new op lower bound from 7333381524588949504 to 7333381523961188352. Current Snapshot: MvccSnapshot[applied={T|T < 7333381524588949504}]
I20260925 22:45:48.799770 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.796013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:51255 (local address 127.0.201.131:46873)
0925 22:45:48.796147 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.796151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.796164 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:45:48.796278 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.796283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.796325 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.796426 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:48.796718 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.797488 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.798704 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.798912 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.798992 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.799302 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:48.799362 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:48.799415 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.799545 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.799602 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:45:48.799806 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.795758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:46873 (local address 127.0.201.130:51255)
0925 22:45:48.796082 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:48.796099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:48.796196 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:48.796434 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:48.796437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:48.796444 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:45:48.796576 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.796581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.797624 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.797628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.798589 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:48.798599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.799037 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.799041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:48.799135 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.799183 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:45:48.799446 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:45:48.799452 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:45:48.799524 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:45:48.799596 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
I20260925 22:45:48.818488 1917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:48.818646 1917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:48.820629 1917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8782ae8e93064816ba290722a858a978: Sending DeleteTablet for 3 replicas of tablet 4f56af0413f84cd4aa986f47f716fcd3
I20260925 22:45:48.821107 2042 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f56af0413f84cd4aa986f47f716fcd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:48 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:50392
I20260925 22:45:48.821204 2338 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f56af0413f84cd4aa986f47f716fcd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:48 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:56604
I20260925 22:45:48.821270 2452 tablet_replica.cc:330] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: stopping tablet replica
I20260925 22:45:48.821357 2450 tablet_replica.cc:330] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: stopping tablet replica
I20260925 22:45:48.821368 2452 raft_consensus.cc:2229] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:48.821466 2452 raft_consensus.cc:2258] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:48.821465 2450 raft_consensus.cc:2229] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:48.821643 2450 raft_consensus.cc:2258] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:48.822388 2450 ts_tablet_manager.cc:1905] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:48.823964 2198 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f56af0413f84cd4aa986f47f716fcd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:48 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:60010
I20260925 22:45:48.824647 2452 ts_tablet_manager.cc:1905] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:48.825505 2450 ts_tablet_manager.cc:1918] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:45:48.825599 2450 log.cc:1196] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/4f56af0413f84cd4aa986f47f716fcd3
I20260925 22:45:48.825853 2450 ts_tablet_manager.cc:1939] T 4f56af0413f84cd4aa986f47f716fcd3 P e50104e6aa0b485392abac9b8b6bef7c: Deleting consensus metadata
I20260925 22:45:48.826066 2451 tablet_replica.cc:330] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: stopping tablet replica
I20260925 22:45:48.826193 2451 raft_consensus.cc:2229] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:48.826313 2451 raft_consensus.cc:2258] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:48.827090 2451 ts_tablet_manager.cc:1905] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:48.827337 2452 ts_tablet_manager.cc:1918] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:45:48.827415 2452 log.cc:1196] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/4f56af0413f84cd4aa986f47f716fcd3
I20260925 22:45:48.828099 2452 ts_tablet_manager.cc:1939] T 4f56af0413f84cd4aa986f47f716fcd3 P b0111aaee4004345a2410f5f659f3095: Deleting consensus metadata
I20260925 22:45:48.828809 1905 catalog_manager.cc:4841] TS b0111aaee4004345a2410f5f659f3095 (127.0.201.129:38705): tablet 4f56af0413f84cd4aa986f47f716fcd3 (table test-table [id=6040a05b960340eca25579b91dea883a]) successfully deleted
I20260925 22:45:48.830120 2451 ts_tablet_manager.cc:1918] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:45:48.830209 2451 log.cc:1196] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/4f56af0413f84cd4aa986f47f716fcd3
I20260925 22:45:48.830456 2451 ts_tablet_manager.cc:1939] T 4f56af0413f84cd4aa986f47f716fcd3 P cf6abf67ad4a45fd87ea8a252ac651d8: Deleting consensus metadata
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[1876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376348, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:45:48.831156 1905 catalog_manager.cc:4841] TS cf6abf67ad4a45fd87ea8a252ac651d8 (127.0.201.130:38425): tablet 4f56af0413f84cd4aa986f47f716fcd3 (table test-table [id=6040a05b960340eca25579b91dea883a]) successfully deleted
I20260925 22:45:48.832145 1905 catalog_manager.cc:4841] TS e50104e6aa0b485392abac9b8b6bef7c (127.0.201.131:46873): tablet 4f56af0413f84cd4aa986f47f716fcd3 (table test-table [id=6040a05b960340eca25579b91dea883a]) successfully deleted
I20260925 22:45:48.835265 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.822469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.0.201.129:38705)
0925 22:45:48.822626 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.822631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.823087 (+ 456us) server_negotiation.cc:407] Connection header received
0925 22:45:48.823199 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.823202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.823253 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.823336 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:48.824315 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.825064 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.826681 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.826985 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.827075 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.831396 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:48.831406 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:48.831422 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:48.831450 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:48.833388 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.833852 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.833865 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.833866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.833952 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.834256 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.834261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.834263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.834586 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:48.834815 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.834982 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.835106 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260925 22:45:48.835772 2117 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50396
I20260925 22:45:48.846179 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:48.837452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.0.201.190:37137)
0925 22:45:48.837577 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:48.837580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:48.837643 (+ 63us) server_negotiation.cc:407] Connection header received
0925 22:45:48.837767 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:48.837769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:48.837807 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:48.837871 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:48.838626 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.839141 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.840304 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:48.840646 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:48.840785 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:48.842570 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:48.842573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:48.842576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:48.842602 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:48.844475 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.844926 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.844931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.844933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.845024 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:48.845317 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:48.845320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:48.845322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:48.845502 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:48.845652 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:48.845903 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 22:45:48.846022 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 22:45:48.846591 1917 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60738
I20260925 22:45:48.847916 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 1993
I20260925 22:45:48.857724 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 2133
I20260925 22:45:48.864759 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 2273
I20260925 22:45:48.871163 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 1889
2026-09-25T22:45:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2611 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[2489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[2489](info): set up 2 sockets
Sep 25 22:45:48 dist-test-slave-vdtf krb5kdc[2489](info): commencing operation
krb5kdc: starting...
W20260925 22:45:49.919596 806 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:45:50.040768 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:44933
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:44933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:50.151800 2502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:50.152078 2502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:50.152118 2502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:50.156035 2502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:50.156087 2502 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:45:50.156104 2502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:50.156123 2502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:50.156138 2502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:50.156153 2502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:50.161350 2502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:44933
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:44933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:50.162352 2502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:50.163324 2502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:50.171808 2507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:50.171800 2508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:50.171900 2502 server_base.cc:1034] running on GCE node
W20260925 22:45:50.171816 2510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:50.172755 2502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:50.173825 2502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:50.175041 2502 hybrid_clock.cc:648] HybridClock initialized: now 1790376350175023 us; error 31 us; skew 500 ppm
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:50.183327 2502 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:45:50.184337 2502 webserver.cc:460] Webserver started at http://127.0.201.190:41291/ using document root <none> and password file <none>
I20260925 22:45:50.184948 2502 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:50.185051 2502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:50.185286 2502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:50.187343 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "836ee11b65944d13aaba8c92f93e25e5"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "c5af395ae6198107b3aaa0db1fd694dc"
server_key_iv: "48f3d2378ca5f242935168878555d06c"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.187925 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "836ee11b65944d13aaba8c92f93e25e5"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "c5af395ae6198107b3aaa0db1fd694dc"
server_key_iv: "48f3d2378ca5f242935168878555d06c"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.191481 2502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:50.193914 2517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:50.195092 2502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:50.195230 2502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "836ee11b65944d13aaba8c92f93e25e5"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "c5af395ae6198107b3aaa0db1fd694dc"
server_key_iv: "48f3d2378ca5f242935168878555d06c"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.195339 2502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:50.222121 2502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:50.225243 2502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:50.225448 2502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:50.233538 2502 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:44933
I20260925 22:45:50.233562 2579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:44933 every 8 connection(s)
I20260925 22:45:50.234458 2502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:50.236534 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 2502
I20260925 22:45:50.236665 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:45:50.236913 806 external_mini_cluster.cc:1428] Setting key ef851370cc33ab2d99808af135fcbef6
I20260925 22:45:50.237397 2580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:50.243328 2580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5: Bootstrap starting.
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:50.245712 2580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:50.246579 2580 log.cc:824] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:50.248585 2580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5: No bootstrap required, opened a new log
I20260925 22:45:50.251163 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.238378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49140 (local address 127.0.201.190:44933)
0925 22:45:50.238807 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:50.238822 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:50.238856 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:45:50.239491 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:50.239508 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:50.239915 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:50.240259 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:50.241030 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.242001 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.242847 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.243338 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.243520 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.246222 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:50.246226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:50.246239 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:50.246263 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:50.248507 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.249016 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.249021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.249024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.249136 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.249420 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.249422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.249423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.249756 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:50.249889 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:50.250183 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 22:45:50.250455 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20260925 22:45:50.251969 2580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836ee11b65944d13aaba8c92f93e25e5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44933 } }
I20260925 22:45:50.252202 2580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:50.252291 2580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836ee11b65944d13aaba8c92f93e25e5, State: Initialized, Role: FOLLOWER
I20260925 22:45:50.252698 2580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836ee11b65944d13aaba8c92f93e25e5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44933 } }
I20260925 22:45:50.252848 2580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:50.252934 2580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:50.253059 2580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:50.253882 2580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836ee11b65944d13aaba8c92f93e25e5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44933 } }
I20260925 22:45:50.254221 2580 leader_election.cc:302] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 836ee11b65944d13aaba8c92f93e25e5; no voters:
I20260925 22:45:50.254521 2580 leader_election.cc:288] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:50.254686 2585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:50.254904 2585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [term 1 LEADER]: Becoming Leader. State: Replica: 836ee11b65944d13aaba8c92f93e25e5, State: Running, Role: LEADER
I20260925 22:45:50.255178 2585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836ee11b65944d13aaba8c92f93e25e5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44933 } }
I20260925 22:45:50.255738 2580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:50.257079 2586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "836ee11b65944d13aaba8c92f93e25e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836ee11b65944d13aaba8c92f93e25e5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44933 } } }
I20260925 22:45:50.257115 2587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 836ee11b65944d13aaba8c92f93e25e5. Latest consensus state: current_term: 1 leader_uuid: "836ee11b65944d13aaba8c92f93e25e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836ee11b65944d13aaba8c92f93e25e5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44933 } } }
I20260925 22:45:50.257227 2586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:50.257227 2587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:50.257596 2596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:50.260596 2596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:50.265609 2596 catalog_manager.cc:1347] Generated new cluster ID: 356337e379734e97b2331a5e6897f4b5
I20260925 22:45:50.265671 2596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:50.278219 2596 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:50.279414 2596 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:50.289238 2596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5: Generated new TSK 0
I20260925 22:45:50.290052 2596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:45:50.331503 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44933
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:50.440537 2606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:50.440838 2606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:50.440922 2606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:50.444924 2606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:50.445019 2606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:50.445089 2606 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:45:50.445138 2606 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:45:50.445199 2606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:45:50.450258 2606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.201.190:44933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:50.451243 2606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:50.452257 2606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:50.462625 2611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:50.462747 2606 server_base.cc:1034] running on GCE node
W20260925 22:45:50.462625 2614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:50.462625 2612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:50.463243 2606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:50.463954 2606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:50.465147 2606 hybrid_clock.cc:648] HybridClock initialized: now 1790376350465132 us; error 27 us; skew 500 ppm
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:50.467904 2606 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:45:50.468869 2606 webserver.cc:460] Webserver started at http://127.0.201.129:44229/ using document root <none> and password file <none>
I20260925 22:45:50.469420 2606 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:50.469524 2606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:50.469735 2606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:50.471764 2606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "1960268384784152be5c4612b4d42fe0"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "76a0260d93f55e816480c5ef7849895b"
server_key_iv: "0b4affd304a486e8fe1695266b1184b0"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.472328 2606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "1960268384784152be5c4612b4d42fe0"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "76a0260d93f55e816480c5ef7849895b"
server_key_iv: "0b4affd304a486e8fe1695266b1184b0"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.475993 2606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260925 22:45:50.478474 2621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:50.479642 2606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:50.479821 2606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "1960268384784152be5c4612b4d42fe0"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "76a0260d93f55e816480c5ef7849895b"
server_key_iv: "0b4affd304a486e8fe1695266b1184b0"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.479985 2606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:50.501407 2606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:50.504905 2606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:50.505137 2606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:50.506592 2606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:50.506675 2606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:50.506742 2606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:50.506780 2606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:50.517745 2606 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:35767
I20260925 22:45:50.517805 2736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:35767 every 8 connection(s)
I20260925 22:45:50.518652 2606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:50.520563 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.507527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:40077 (local address 127.0.201.190:44933)
0925 22:45:50.507651 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:50.507655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:50.508843 (+ 1188us) server_negotiation.cc:407] Connection header received
0925 22:45:50.509957 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:50.509960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:50.510022 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:50.510105 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:50.511429 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.512238 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.513465 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.513718 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.513852 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.516650 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:50.516653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:50.516656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:50.516682 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:50.518261 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.518820 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.518823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.518825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.518909 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.519475 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.519479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.519483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.519731 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:50.519831 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:50.520330 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 22:45:50.520443 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:45:50.521349 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.508135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44933 (local address 127.0.201.129:40077)
0925 22:45:50.508678 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:50.508721 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:50.509710 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:50.510369 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:50.510376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:50.510785 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:50.511267 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.511281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.512364 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.512369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.513321 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.513330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.513909 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.513914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.514158 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.514826 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:50.514844 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:50.516477 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:50.518389 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.518396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.518411 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.518727 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.519011 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.519016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.519018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.519215 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.519829 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:50.519834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:50.520071 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 22:45:50.520236 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 22:45:50.520495 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":123,"threads_started":1}
I20260925 22:45:50.522670 2737 heartbeater.cc:347] Connected to a master server at 127.0.201.190:44933
I20260925 22:45:50.522832 2737 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:50.523311 2737 heartbeater.cc:511] Master 127.0.201.190:44933 requested a full tablet report, sending...
I20260925 22:45:50.525012 2533 ts_manager.cc:194] Registered new tserver with Master: 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767)
I20260925 22:45:50.526154 2533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:40077
I20260925 22:45:50.528098 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 2606
I20260925 22:45:50.528219 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:50.528510 806 external_mini_cluster.cc:1428] Setting key 5c8a0c27b9df74ab4eaaefc55263a371
I20260925 22:45:50.533864 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.524751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44933 (local address 127.0.201.129:41957)
0925 22:45:50.525020 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:50.525033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:50.525137 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:50.526959 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:50.526962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:50.527176 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:50.527269 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.527274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.528095 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.528098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.528825 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.528834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.529381 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.529384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.529483 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.530043 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:50.530054 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:50.530356 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:50.532439 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.532442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.532443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.532644 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.532939 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.532941 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.532943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.533015 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.533486 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:50.533488 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:50.533558 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 22:45:50.533635 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:45:50.533736 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260925 22:45:50.534155 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.524845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:41957 (local address 127.0.201.190:44933)
0925 22:45:50.526760 (+ 1915us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:50.526767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:50.526781 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:45:50.526816 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:50.526818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:50.526850 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:50.526942 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:50.527392 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.527971 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.528953 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.529258 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.529381 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.530521 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:50.530524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:50.530527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:50.530549 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:50.532316 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.532742 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.532748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.532753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.532844 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.533119 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.533124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.533129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.533383 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:50.533497 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:50.533731 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 22:45:50.533862 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1850}
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:45:50.559165 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44933
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:50.669380 2746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:50.669639 2746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:50.669685 2746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:50.673655 2746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:50.673708 2746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:50.673748 2746 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:45:50.673820 2746 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:45:50.673878 2746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:45:50.678932 2746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.201.190:44933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:50.679991 2746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:50.680989 2746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:50.690423 2752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:50.690438 2751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:50.690656 2746 server_base.cc:1034] running on GCE node
W20260925 22:45:50.690778 2754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:50.691138 2746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:50.691892 2746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:50.693094 2746 hybrid_clock.cc:648] HybridClock initialized: now 1790376350693066 us; error 37 us; skew 500 ppm
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:50.695864 2746 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:45:50.696820 2746 webserver.cc:460] Webserver started at http://127.0.201.130:36289/ using document root <none> and password file <none>
I20260925 22:45:50.697400 2746 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:50.697503 2746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:50.697721 2746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:50.699976 2746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "a717d14b129f4dae83fe0984a7ba4cdb"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "e157e08f8f4021cff4dbf51c9e29773a"
server_key_iv: "c1e0651276dbd1505275e8e9716e21f2"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.700587 2746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "a717d14b129f4dae83fe0984a7ba4cdb"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "e157e08f8f4021cff4dbf51c9e29773a"
server_key_iv: "c1e0651276dbd1505275e8e9716e21f2"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.704339 2746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:45:50.706697 2761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:50.708055 2746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:45:50.708189 2746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "a717d14b129f4dae83fe0984a7ba4cdb"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "e157e08f8f4021cff4dbf51c9e29773a"
server_key_iv: "c1e0651276dbd1505275e8e9716e21f2"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.708304 2746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:50.736263 2746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:50.739202 2746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:50.739404 2746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:50.740885 2746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:50.740964 2746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:50.741036 2746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:50.741073 2746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:50.751793 2746 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:45699
I20260925 22:45:50.751994 2876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:45699 every 8 connection(s)
I20260925 22:45:50.752838 2746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:50.755571 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 2746
I20260925 22:45:50.755646 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.741963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:48611 (local address 127.0.201.190:44933)
0925 22:45:50.742107 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:50.742112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:50.743150 (+ 1038us) server_negotiation.cc:407] Connection header received
0925 22:45:50.744467 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:50.744471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:50.744527 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:50.744590 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:50.746414 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.747324 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.748714 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.749001 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.749130 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.752135 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:50.752138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:50.752141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:50.752167 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:50.753722 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.754258 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.754261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.754262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.754338 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.754641 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.754643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.754644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.754808 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:50.754880 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:50.755304 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 22:45:50.755465 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:45:50.755694 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:50.755964 806 external_mini_cluster.cc:1428] Setting key cb7dcaa5a56a0be5def1df36b4035d10
I20260925 22:45:50.756466 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.742470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44933 (local address 127.0.201.130:48611)
0925 22:45:50.742992 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:50.743034 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:50.744204 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:50.745027 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:50.745034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:50.745536 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:50.746225 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.746246 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.747461 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.747465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.748584 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.748593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.749165 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.749168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.749344 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.749994 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:50.750017 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:50.751936 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:50.753844 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.753849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.753863 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.754147 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.754431 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.754433 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.754435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.754554 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.754893 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:50.754899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:50.755102 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 22:45:50.755246 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 22:45:50.755484 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
I20260925 22:45:50.757704 2877 heartbeater.cc:347] Connected to a master server at 127.0.201.190:44933
I20260925 22:45:50.757862 2877 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:50.758356 2877 heartbeater.cc:511] Master 127.0.201.190:44933 requested a full tablet report, sending...
I20260925 22:45:50.759408 2534 ts_manager.cc:194] Registered new tserver with Master: a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130:45699)
I20260925 22:45:50.760270 2534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:48611
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:45:50.768464 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.760155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44933 (local address 127.0.201.130:54405)
0925 22:45:50.760666 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:50.760685 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:50.760989 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:50.761299 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:50.761302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:50.761507 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:50.761659 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.761668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.762783 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.762786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.763339 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.763346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.764044 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.764047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.764117 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.764662 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:50.764680 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:50.764993 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:50.767033 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.767036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.767037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.767223 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.767491 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.767494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.767495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.767568 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.768040 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:50.768042 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:50.768116 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 22:45:50.768204 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:45:50.768307 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":7,"thread_start_us":79,"threads_started":1}
I20260925 22:45:50.768471 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.760248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:54405 (local address 127.0.201.190:44933)
0925 22:45:50.760391 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:50.760395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:50.760780 (+ 385us) server_negotiation.cc:407] Connection header received
0925 22:45:50.761102 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:50.761108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:50.761164 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:50.761252 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:50.761787 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.762639 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.763468 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.763913 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.764042 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.765139 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:50.765145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:50.765149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:50.765176 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:50.766907 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.767320 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.767323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.767324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.767395 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.767651 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.767653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.767654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.767948 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:50.768028 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:50.768281 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 22:45:50.768367 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:45:50.787619 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44933
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:50.899501 2886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:50.899848 2886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:50.900019 2886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:50.904138 2886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:50.904222 2886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:50.904294 2886 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:45:50.904340 2886 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:45:50.904404 2886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:45:50.909596 2886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.201.190:44933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:50.910681 2886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:50.911803 2886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:50.921443 2892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:50.921443 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:50.921507 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:50.921736 2886 server_base.cc:1034] running on GCE node
I20260925 22:45:50.922199 2886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:50.922900 2886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:50.924110 2886 hybrid_clock.cc:648] HybridClock initialized: now 1790376350924088 us; error 39 us; skew 500 ppm
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:50.926836 2886 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:45:50.927796 2886 webserver.cc:460] Webserver started at http://127.0.201.131:35613/ using document root <none> and password file <none>
I20260925 22:45:50.928365 2886 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:50.928462 2886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:50.928702 2886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:50.930698 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "df47a9c9812f41c6b7f04092194db05a"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "03993d50e17a8c298b4de9249eb1b570"
server_key_iv: "2fc95b399a8ffffa751423eaea4c430f"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.931259 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "df47a9c9812f41c6b7f04092194db05a"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "03993d50e17a8c298b4de9249eb1b570"
server_key_iv: "2fc95b399a8ffffa751423eaea4c430f"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.934980 2886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:45:50.937381 2901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:50.938544 2886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 22:45:50.938683 2886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "df47a9c9812f41c6b7f04092194db05a"
format_stamp: "Formatted at 2026-09-25 22:45:50 on dist-test-slave-vdtf"
server_key: "03993d50e17a8c298b4de9249eb1b570"
server_key_iv: "2fc95b399a8ffffa751423eaea4c430f"
server_key_version: "encryptionkey@0"
I20260925 22:45:50.938793 2886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:50.951759 2886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:50.954798 2886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:50.954994 2886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:50.956480 2886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:50.956557 2886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:50.956660 2886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:50.956686 2886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:50.967077 2886 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:46365
I20260925 22:45:50.967104 3016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:46365 every 8 connection(s)
I20260925 22:45:50.967952 2886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:50.971101 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.957252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:55745 (local address 127.0.201.190:44933)
0925 22:45:50.957363 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:50.957368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:50.958432 (+ 1064us) server_negotiation.cc:407] Connection header received
0925 22:45:50.959577 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:50.959583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:50.959657 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:50.959774 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:50.961456 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.962288 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.964138 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.964384 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.964466 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.967440 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:50.967443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:50.967446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:50.967477 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:50.969079 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.969627 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.969632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.969634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.969717 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.970063 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.970068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.970070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.970253 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:50.970365 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:50.970839 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 22:45:50.970948 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 22:45:50.971917 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.957813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44933 (local address 127.0.201.131:55745)
0925 22:45:50.958236 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:50.958322 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:50.959320 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:50.960112 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:50.960123 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:50.960563 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:50.961279 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.961301 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.962423 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.962428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.964005 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.964017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.964551 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.964556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.964773 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.965416 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:50.965443 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:50.967275 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:50.969202 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.969209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.969224 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.969520 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.969815 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.969819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.969821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.969955 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.970362 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:50.970368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:50.970601 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 22:45:50.970798 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 22:45:50.971095 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260925 22:45:50.973006 3017 heartbeater.cc:347] Connected to a master server at 127.0.201.190:44933
I20260925 22:45:50.973213 3017 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:50.973750 3017 heartbeater.cc:511] Master 127.0.201.190:44933 requested a full tablet report, sending...
I20260925 22:45:50.973788 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 2886
I20260925 22:45:50.973922 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:50.974290 806 external_mini_cluster.cc:1428] Setting key 29b3177acb50a603a167c30eb49b9f5a
I20260925 22:45:50.974822 2533 ts_manager.cc:194] Registered new tserver with Master: df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365)
I20260925 22:45:50.975561 2533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:55745
I20260925 22:45:50.977638 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:45:50.982731 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.975382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:42209 (local address 127.0.201.190:44933)
0925 22:45:50.975491 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:50.975495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:50.975739 (+ 244us) server_negotiation.cc:407] Connection header received
0925 22:45:50.975881 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:50.975884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:50.975919 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:50.975983 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:50.976517 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.977234 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.978256 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.978466 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.978589 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.979630 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:50.979633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:50.979635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:50.979657 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:50.981131 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.981566 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.981571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.981573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.981650 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:50.981940 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:50.981946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:50.981948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:50.982117 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:50.982227 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:50.982480 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 22:45:50.982587 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:45:50.982739 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.975263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44933 (local address 127.0.201.131:42209)
0925 22:45:50.975645 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:50.975659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:50.975799 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:50.976096 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:50.976099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:50.976287 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:50.976396 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.976402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.977358 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.977361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.978136 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:50.978145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.978592 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.978596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:50.978679 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:50.979195 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:50.979206 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:50.979497 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:50.981268 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.981271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.981273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.981464 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.981751 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:50.981755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:50.981757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:50.981843 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:50.982217 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:50.982220 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:50.982306 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 22:45:50.982419 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:45:50.982552 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":100,"threads_started":1}
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:45:50 dist-test-slave-vdtf krb5kdc[2489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:51.003147 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:50.995263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.0.201.190:44933)
0925 22:45:50.995382 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:50.995385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:50.995396 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:45:50.995428 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:50.995430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:50.995463 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:50.995539 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:50.996236 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.996726 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.997510 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:50.997708 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:50.997783 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.000176 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:51.000179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:51.000181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:51.000202 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:51.001598 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:51.001938 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:51.001940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:51.001941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:51.002009 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:51.002309 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:51.002312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:51.002313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:51.002500 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:51.002608 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.002796 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.002882 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:45:51.006187 2533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:45:51.009214 2533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:51.023341 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.017626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:35767 (local address 127.0.0.1:59944)
0925 22:45:51.018337 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.018375 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.018469 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.019219 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.019223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.019252 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:51.019359 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.019364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.020658 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.020662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.021967 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.021976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.022829 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.022832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.022936 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.022971 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.023078 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.023197 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":50,"spinlock_wait_cycles":4224,"thread_start_us":106,"threads_started":1}
I20260925 22:45:51.023339 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.017786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.0.201.129:35767)
0925 22:45:51.018215 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.018230 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.018593 (+ 363us) server_negotiation.cc:407] Connection header received
0925 22:45:51.018680 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.018685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.018894 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.019052 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:51.019484 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.020518 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.022091 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.022641 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.022853 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.023026 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.023114 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.023203 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":73,"threads_started":1}
I20260925 22:45:51.024514 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.018245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:45699 (local address 127.0.0.1:49958)
0925 22:45:51.018488 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.018511 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.018585 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.019972 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.019975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.019992 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:51.020070 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.020075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.021453 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.021456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.022826 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.022836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.024128 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.024131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.024214 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.024234 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.024316 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.024381 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":61,"thread_start_us":96,"threads_started":1}
I20260925 22:45:51.024614 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.018949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.0.201.130:45699)
0925 22:45:51.019313 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.019326 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.019350 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:45:51.019427 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.019435 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.019613 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.019823 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:51.020200 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.021329 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.022965 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.023824 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.023919 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.024068 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.024373 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.024480 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":96,"threads_started":1}
I20260925 22:45:51.025025 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.017409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:46365 (local address 127.0.0.1:49460)
0925 22:45:51.018265 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.018319 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.018449 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.019220 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.019227 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.019261 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:51.019388 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.019394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.020776 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.020779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.022036 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.022047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.023262 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.023265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.023366 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.023388 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.024833 (+ 1445us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.024900 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":83,"spinlock_wait_cycles":5760,"thread_start_us":90,"threads_started":1}
I20260925 22:45:51.025043 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.017520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.0.201.131:46365)
0925 22:45:51.017991 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.018003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.018593 (+ 590us) server_negotiation.cc:407] Connection header received
0925 22:45:51.018687 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.018692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.018893 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.019060 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:51.019514 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.020658 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.022172 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.022954 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.023046 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.023203 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.024800 (+ 1597us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.024900 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":97,"threads_started":1}
I20260925 22:45:51.026701 2671 tablet_service.cc:1467] Processing CreateTablet for tablet d10ba0ef41a242538384cb27f29f4062 (DEFAULT_TABLE table=test-table [id=312fe8186a094998855db106fbc45c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:51.027127 2793 tablet_service.cc:1467] Processing CreateTablet for tablet d10ba0ef41a242538384cb27f29f4062 (DEFAULT_TABLE table=test-table [id=312fe8186a094998855db106fbc45c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:51.027916 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10ba0ef41a242538384cb27f29f4062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:51.027628 2951 tablet_service.cc:1467] Processing CreateTablet for tablet d10ba0ef41a242538384cb27f29f4062 (DEFAULT_TABLE table=test-table [id=312fe8186a094998855db106fbc45c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:51.028106 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10ba0ef41a242538384cb27f29f4062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:51.028558 2951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10ba0ef41a242538384cb27f29f4062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:51.035089 3041 tablet_bootstrap.cc:492] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: Bootstrap starting.
I20260925 22:45:51.035804 3042 tablet_bootstrap.cc:492] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: Bootstrap starting.
I20260925 22:45:51.036028 3040 tablet_bootstrap.cc:492] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: Bootstrap starting.
I20260925 22:45:51.037423 3041 tablet_bootstrap.cc:654] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:51.037985 3042 tablet_bootstrap.cc:654] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:51.038005 3040 tablet_bootstrap.cc:654] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:51.038532 3041 log.cc:824] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:51.038802 3042 log.cc:824] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:51.038802 3040 log.cc:824] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:51.040611 3040 tablet_bootstrap.cc:492] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: No bootstrap required, opened a new log
I20260925 22:45:51.040611 3041 tablet_bootstrap.cc:492] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: No bootstrap required, opened a new log
I20260925 22:45:51.040800 3040 ts_tablet_manager.cc:1397] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:45:51.040800 3041 ts_tablet_manager.cc:1397] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:45:51.040951 3042 tablet_bootstrap.cc:492] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: No bootstrap required, opened a new log
I20260925 22:45:51.041150 3042 ts_tablet_manager.cc:1397] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 22:45:51.044305 3042 raft_consensus.cc:348] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } }
I20260925 22:45:51.044549 3042 raft_consensus.cc:374] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:51.044600 3042 raft_consensus.cc:729] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a717d14b129f4dae83fe0984a7ba4cdb, State: Initialized, Role: FOLLOWER
I20260925 22:45:51.044721 3040 raft_consensus.cc:348] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } }
I20260925 22:45:51.044721 3041 raft_consensus.cc:348] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } }
I20260925 22:45:51.044965 3041 raft_consensus.cc:374] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:51.044965 3040 raft_consensus.cc:374] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:51.045022 3041 raft_consensus.cc:729] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1960268384784152be5c4612b4d42fe0, State: Initialized, Role: FOLLOWER
I20260925 22:45:51.045022 3040 raft_consensus.cc:729] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df47a9c9812f41c6b7f04092194db05a, State: Initialized, Role: FOLLOWER
I20260925 22:45:51.045075 3042 consensus_queue.cc:260] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } }
I20260925 22:45:51.045457 3040 consensus_queue.cc:260] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } }
I20260925 22:45:51.045455 3041 consensus_queue.cc:260] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } }
I20260925 22:45:51.045811 2877 heartbeater.cc:503] Master 127.0.201.190:44933 was elected leader, sending a full tablet report...
I20260925 22:45:51.046090 3042 ts_tablet_manager.cc:1428] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:45:51.046325 3041 ts_tablet_manager.cc:1428] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:45:51.046626 2737 heartbeater.cc:503] Master 127.0.201.190:44933 was elected leader, sending a full tablet report...
I20260925 22:45:51.047413 3040 ts_tablet_manager.cc:1428] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:45:51.047757 3017 heartbeater.cc:503] Master 127.0.201.190:44933 was elected leader, sending a full tablet report...
I20260925 22:45:51.205081 3047 raft_consensus.cc:482] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:51.205329 3047 raft_consensus.cc:504] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } }
I20260925 22:45:51.206475 3047 leader_election.cc:288] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365), a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130:45699)
I20260925 22:45:51.210757 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.207174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:45699 (local address 127.0.201.129:53453)
0925 22:45:51.207461 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.207475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.207574 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.207845 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.207848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.207862 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:51.208009 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.208016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.208758 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.208760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.209677 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.209686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.210326 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.210330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.210424 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.210450 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.210520 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.210569 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":71,"thread_start_us":91,"threads_started":1}
I20260925 22:45:51.210757 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.207297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:53453 (local address 127.0.201.130:45699)
0925 22:45:51.207440 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.207444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.207558 (+ 114us) server_negotiation.cc:407] Connection header received
0925 22:45:51.207700 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.207703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.207748 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.207830 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:51.208124 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.208643 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.209806 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.210203 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.210331 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.210386 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.210557 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.210638 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:45:51.210819 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.206788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:46365 (local address 127.0.201.129:37513)
0925 22:45:51.207140 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.207156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.207290 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.207696 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.207699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.207719 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:51.207857 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.207864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.208685 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.208691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.209692 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.209701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.210380 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.210384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.210493 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.210543 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.210620 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.210678 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":80,"thread_start_us":97,"threads_started":1}
I20260925 22:45:51.210844 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.207010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:37513 (local address 127.0.201.131:46365)
0925 22:45:51.207175 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.207180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.207241 (+ 61us) server_negotiation.cc:407] Connection header received
0925 22:45:51.207476 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.207481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.207552 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.207628 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:51.207986 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.208560 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.209852 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.210223 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.210360 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.210412 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.210656 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.210720 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 22:45:51.211439 2971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10ba0ef41a242538384cb27f29f4062" candidate_uuid: "1960268384784152be5c4612b4d42fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df47a9c9812f41c6b7f04092194db05a" is_pre_election: true
I20260925 22:45:51.211427 2831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10ba0ef41a242538384cb27f29f4062" candidate_uuid: "1960268384784152be5c4612b4d42fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" is_pre_election: true
I20260925 22:45:51.211757 2831 raft_consensus.cc:2454] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1960268384784152be5c4612b4d42fe0 in term 0.
I20260925 22:45:51.211757 2971 raft_consensus.cc:2454] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1960268384784152be5c4612b4d42fe0 in term 0.
I20260925 22:45:51.212250 2623 leader_election.cc:302] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1960268384784152be5c4612b4d42fe0, a717d14b129f4dae83fe0984a7ba4cdb; no voters:
I20260925 22:45:51.212496 3047 raft_consensus.cc:2781] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:51.212580 3047 raft_consensus.cc:482] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:51.212625 3047 raft_consensus.cc:3037] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:51.213644 3047 raft_consensus.cc:504] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } }
I20260925 22:45:51.213977 3047 leader_election.cc:288] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [CANDIDATE]: Term 1 election: Requested vote from peers df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365), a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130:45699)
I20260925 22:45:51.214321 2831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10ba0ef41a242538384cb27f29f4062" candidate_uuid: "1960268384784152be5c4612b4d42fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a717d14b129f4dae83fe0984a7ba4cdb"
I20260925 22:45:51.214321 2971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10ba0ef41a242538384cb27f29f4062" candidate_uuid: "1960268384784152be5c4612b4d42fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df47a9c9812f41c6b7f04092194db05a"
I20260925 22:45:51.214447 2971 raft_consensus.cc:3037] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:51.214442 2831 raft_consensus.cc:3037] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:51.215819 2831 raft_consensus.cc:2454] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1960268384784152be5c4612b4d42fe0 in term 1.
I20260925 22:45:51.216030 2971 raft_consensus.cc:2454] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1960268384784152be5c4612b4d42fe0 in term 1.
I20260925 22:45:51.216117 2623 leader_election.cc:302] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1960268384784152be5c4612b4d42fe0, a717d14b129f4dae83fe0984a7ba4cdb; no voters:
I20260925 22:45:51.216259 3047 raft_consensus.cc:2781] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:51.216467 3047 raft_consensus.cc:686] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 1 LEADER]: Becoming Leader. State: Replica: 1960268384784152be5c4612b4d42fe0, State: Running, Role: LEADER
I20260925 22:45:51.216835 3047 consensus_queue.cc:237] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } }
W20260925 22:45:51.218874 3018 tablet.cc:2369] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:45:51.220233 2533 catalog_manager.cc:5495] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1960268384784152be5c4612b4d42fe0 (127.0.201.129). New cstate: current_term: 1 leader_uuid: "1960268384784152be5c4612b4d42fe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } health_report { overall_health: HEALTHY } } }
W20260925 22:45:51.254382 2878 tablet.cc:2369] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:45:51.270121 2738 tablet.cc:2369] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 22:45:51 dist-test-slave-vdtf krb5kdc[2489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:51.278494 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.269433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.0.201.190:44933)
0925 22:45:51.269596 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.269603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.269782 (+ 179us) server_negotiation.cc:407] Connection header received
0925 22:45:51.270001 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.270003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.270041 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.270138 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:51.270837 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.271404 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.272369 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.272660 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.272782 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.275229 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:51.275233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:51.275236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:51.275257 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:51.276814 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:51.277286 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:51.277292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:51.277294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:51.277368 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:51.277650 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:51.277653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:51.277655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:51.277817 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:51.277924 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.278206 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.278354 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":46}
I20260925 22:45:51.278494 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.269348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44933 (local address 127.0.0.1:49154)
0925 22:45:51.269684 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.269697 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.269842 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.270157 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.270159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.270605 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:51.270706 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.270712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.271526 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.271531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.272228 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.272237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.272779 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.272783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.272896 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.273552 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:51.273565 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:51.275087 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:51.276942 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:51.276952 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:51.276959 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:51.277184 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:51.277463 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:51.277468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:51.277470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:51.277555 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:51.277931 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:51.277940 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:51.278044 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.278163 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.278305 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":22,"thread_start_us":112,"threads_started":1}
I20260925 22:45:51.282382 2533 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:49154:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:45:51.283135 2533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:51.287644 2793 tablet_service.cc:1467] Processing CreateTablet for tablet 6621328ef2b84a37b94bd130ec56fb31 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5ce0f149e8442ef8ee31f44a04b877d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:45:51.287631 2671 tablet_service.cc:1467] Processing CreateTablet for tablet 6621328ef2b84a37b94bd130ec56fb31 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5ce0f149e8442ef8ee31f44a04b877d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:45:51.288427 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6621328ef2b84a37b94bd130ec56fb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:51.288439 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6621328ef2b84a37b94bd130ec56fb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:51.290800 3041 tablet_bootstrap.cc:492] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: Bootstrap starting.
I20260925 22:45:51.290807 3042 tablet_bootstrap.cc:492] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: Bootstrap starting.
I20260925 22:45:51.291746 3042 tablet_bootstrap.cc:654] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:51.291468 2951 tablet_service.cc:1467] Processing CreateTablet for tablet 6621328ef2b84a37b94bd130ec56fb31 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5ce0f149e8442ef8ee31f44a04b877d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:45:51.292153 2951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6621328ef2b84a37b94bd130ec56fb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:51.292241 3041 tablet_bootstrap.cc:654] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:51.293006 3042 tablet_bootstrap.cc:492] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: No bootstrap required, opened a new log
I20260925 22:45:51.293093 3042 ts_tablet_manager.cc:1397] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 22:45:51.293525 3041 tablet_bootstrap.cc:492] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: No bootstrap required, opened a new log
I20260925 22:45:51.293604 3041 ts_tablet_manager.cc:1397] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:45:51.293876 3042 raft_consensus.cc:348] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.293998 3042 raft_consensus.cc:374] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:51.294044 3042 raft_consensus.cc:729] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a717d14b129f4dae83fe0984a7ba4cdb, State: Initialized, Role: FOLLOWER
I20260925 22:45:51.294027 3041 raft_consensus.cc:348] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.294184 3041 raft_consensus.cc:374] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:51.294222 3041 raft_consensus.cc:729] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1960268384784152be5c4612b4d42fe0, State: Initialized, Role: FOLLOWER
I20260925 22:45:51.294168 3042 consensus_queue.cc:260] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.294361 3041 consensus_queue.cc:260] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.294440 3042 ts_tablet_manager.cc:1428] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260925 22:45:51.294605 3047 tablet_replica.cc:439] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } }
I20260925 22:45:51.294606 3046 tablet_replica.cc:439] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } }
I20260925 22:45:51.294724 3047 tablet_replica.cc:442] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:51.294724 3046 tablet_replica.cc:442] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:51.294926 3041 ts_tablet_manager.cc:1428] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:45:51.295357 3040 tablet_bootstrap.cc:492] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: Bootstrap starting.
I20260925 22:45:51.296425 3040 tablet_bootstrap.cc:654] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:51.297609 3040 tablet_bootstrap.cc:492] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: No bootstrap required, opened a new log
I20260925 22:45:51.297721 3040 ts_tablet_manager.cc:1397] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:51.298115 3040 raft_consensus.cc:348] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.298202 3040 raft_consensus.cc:374] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:51.298223 3040 raft_consensus.cc:729] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df47a9c9812f41c6b7f04092194db05a, State: Initialized, Role: FOLLOWER
I20260925 22:45:51.298355 3040 consensus_queue.cc:260] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.298637 3040 ts_tablet_manager.cc:1428] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:45:51.298609 3048 tablet_replica.cc:439] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } }
I20260925 22:45:51.298727 3048 tablet_replica.cc:442] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:51.458230 3046 raft_consensus.cc:482] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:51.458372 3046 raft_consensus.cc:504] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.459489 3046 leader_election.cc:288] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767), df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365)
I20260925 22:45:51.463461 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.460014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:41879 (local address 127.0.201.131:46365)
0925 22:45:51.460154 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.460158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.460229 (+ 71us) server_negotiation.cc:407] Connection header received
0925 22:45:51.460379 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.460382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.460428 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.460516 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:51.460801 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.461306 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.462474 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.462867 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.463005 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.463062 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.463266 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.463326 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:45:51.463480 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.459789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:51267 (local address 127.0.201.129:35767)
0925 22:45:51.459945 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.459949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.460221 (+ 272us) server_negotiation.cc:407] Connection header received
0925 22:45:51.460383 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.460386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.460433 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.460525 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:51.460788 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.461326 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.462484 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.462871 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.463020 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.463075 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.463290 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.463354 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:45:51.463507 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.459740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:46365 (local address 127.0.201.130:41879)
0925 22:45:51.460161 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.460177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.460275 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.460531 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.460534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.460549 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:51.460693 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.460697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.461430 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.461433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.462347 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.462355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.463004 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.463014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.463132 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.463160 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.463250 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.463318 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":44,"spinlock_wait_cycles":8960,"thread_start_us":89,"threads_started":1}
I20260925 22:45:51.463527 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.459712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:35767 (local address 127.0.201.130:51267)
0925 22:45:51.460114 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.460133 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.460266 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.460536 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.460539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.460566 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:51.460679 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.460684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.461477 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.461481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.462353 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.462361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.463016 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.463019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.463168 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.463184 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.463276 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.463342 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":81,"spinlock_wait_cycles":2816,"thread_start_us":134,"threads_started":1}
I20260925 22:45:51.464051 2971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6621328ef2b84a37b94bd130ec56fb31" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df47a9c9812f41c6b7f04092194db05a" is_pre_election: true
I20260925 22:45:51.464069 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6621328ef2b84a37b94bd130ec56fb31" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1960268384784152be5c4612b4d42fe0" is_pre_election: true
I20260925 22:45:51.464191 2971 raft_consensus.cc:2454] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 0.
I20260925 22:45:51.464298 2691 raft_consensus.cc:2454] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 0.
I20260925 22:45:51.464563 2763 leader_election.cc:302] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a717d14b129f4dae83fe0984a7ba4cdb, df47a9c9812f41c6b7f04092194db05a; no voters:
I20260925 22:45:51.464828 3046 raft_consensus.cc:2781] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:51.464887 3046 raft_consensus.cc:482] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:51.464926 3046 raft_consensus.cc:3037] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:51.465866 3046 raft_consensus.cc:504] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.466172 3046 leader_election.cc:288] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 election: Requested vote from peers 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767), df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365)
I20260925 22:45:51.466437 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6621328ef2b84a37b94bd130ec56fb31" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1960268384784152be5c4612b4d42fe0"
I20260925 22:45:51.466490 2971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6621328ef2b84a37b94bd130ec56fb31" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df47a9c9812f41c6b7f04092194db05a"
I20260925 22:45:51.466580 2691 raft_consensus.cc:3037] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:51.466602 2971 raft_consensus.cc:3037] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:51.467427 2971 raft_consensus.cc:2454] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 1.
I20260925 22:45:51.467589 2691 raft_consensus.cc:2454] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 1.
I20260925 22:45:51.467703 2763 leader_election.cc:302] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a717d14b129f4dae83fe0984a7ba4cdb, df47a9c9812f41c6b7f04092194db05a; no voters:
I20260925 22:45:51.467844 3046 raft_consensus.cc:2781] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:51.468043 3046 raft_consensus.cc:686] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 LEADER]: Becoming Leader. State: Replica: a717d14b129f4dae83fe0984a7ba4cdb, State: Running, Role: LEADER
I20260925 22:45:51.468369 3046 consensus_queue.cc:237] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.470821 3064 tablet_replica.cc:439] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: TxnStatusTablet state changed. Reason: New leader a717d14b129f4dae83fe0984a7ba4cdb. Latest consensus state: current_term: 1 leader_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } }
I20260925 22:45:51.470969 3064 tablet_replica.cc:442] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:45:51.471314 3066 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:45:51.471534 3066 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:45:51.472033 2533 catalog_manager.cc:5495] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb reported cstate change: term changed from 0 to 1, leader changed from <none> to a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130). New cstate: current_term: 1 leader_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260925 22:45:51.486771 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.482434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:45699 (local address 127.0.0.1:49966)
0925 22:45:51.482577 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.482596 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.482739 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.483134 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.483137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.483145 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:45:51.483269 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.483275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.484313 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.484319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.484997 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.485006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.485561 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.485563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.485630 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.485673 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:45:51.486425 (+ 752us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:45:51.486439 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.486564 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.486650 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 22:45:51.486771 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.482525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.0.201.130:45699)
0925 22:45:51.482636 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.482640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.482655 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:45:51.482920 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.482924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.482995 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.483105 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:51.483380 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.484159 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.485117 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.485447 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.485550 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.485808 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:51.486282 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:51.486392 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.486588 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.486638 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:45:51.491744 2691 raft_consensus.cc:1261] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Refusing update from remote peer a717d14b129f4dae83fe0984a7ba4cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:51.491770 2971 raft_consensus.cc:1261] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Refusing update from remote peer a717d14b129f4dae83fe0984a7ba4cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:51.492390 3064 consensus_queue.cc:1035] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:51.492580 3046 consensus_queue.cc:1035] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:51.496146 3047 tablet_replica.cc:439] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: TxnStatusTablet state changed. Reason: New leader a717d14b129f4dae83fe0984a7ba4cdb. Latest consensus state: current_term: 1 leader_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } }
I20260925 22:45:51.496255 3047 tablet_replica.cc:442] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:51.496413 3048 tablet_replica.cc:439] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: TxnStatusTablet state changed. Reason: New leader a717d14b129f4dae83fe0984a7ba4cdb. Latest consensus state: current_term: 1 leader_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } }
I20260925 22:45:51.496515 3048 tablet_replica.cc:442] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:51.497874 3046 tablet_replica.cc:439] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } }
I20260925 22:45:51.497983 3046 tablet_replica.cc:442] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:45:51.499166 3047 tablet_replica.cc:439] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } }
I20260925 22:45:51.499265 3047 tablet_replica.cc:442] T 6621328ef2b84a37b94bd130ec56fb31 P 1960268384784152be5c4612b4d42fe0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:51.501564 3046 tablet_replica.cc:439] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } }
I20260925 22:45:51.501673 3046 tablet_replica.cc:442] T 6621328ef2b84a37b94bd130ec56fb31 P a717d14b129f4dae83fe0984a7ba4cdb: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:45:51.502514 3048 tablet_replica.cc:439] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } } }
I20260925 22:45:51.502632 3048 tablet_replica.cc:442] T 6621328ef2b84a37b94bd130ec56fb31 P df47a9c9812f41c6b7f04092194db05a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:45:51.516561 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.512796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.0.201.129:35767)
0925 22:45:51.512906 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.512911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.513095 (+ 184us) server_negotiation.cc:407] Connection header received
0925 22:45:51.513281 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.513283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.513336 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.513451 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:51.513707 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.514249 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.515136 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.515353 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.515472 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.515770 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:51.516182 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:51.516313 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.516373 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.516421 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 22:45:51.528993 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.525225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:48383 (local address 127.0.201.130:45699)
0925 22:45:51.525348 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.525353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.525608 (+ 255us) server_negotiation.cc:407] Connection header received
0925 22:45:51.525762 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.525765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.525819 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.525906 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:51.526162 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.526673 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.527551 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.527920 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.528027 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.528492 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:51.528581 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:51.528638 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.528838 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.528889 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:45:51.529430 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.525129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:45699 (local address 127.0.201.129:48383)
0925 22:45:51.525509 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.525521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.525653 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.525916 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.525918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.525926 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:45:51.526056 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.526062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.526787 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.526793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.527428 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.527436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.528126 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.528131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.528237 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.528293 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:45:51.528728 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:45:51.528734 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.529206 (+ 472us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.529282 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":82,"threads_started":1}
I20260925 22:45:51.536087 2971 raft_consensus.cc:1261] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Refusing update from remote peer 1960268384784152be5c4612b4d42fe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:51.536264 2831 raft_consensus.cc:1261] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Refusing update from remote peer 1960268384784152be5c4612b4d42fe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:51.536523 3051 consensus_queue.cc:1035] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:51.537151 3051 consensus_queue.cc:1035] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:51.554713 3069 mvcc.cc:205] Tried to move back new op lower bound from 7333381535887577088 to 7333381534586949632. Current Snapshot: MvccSnapshot[applied={T|T < 7333381535887577088}]
I20260925 22:45:51.563464 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.554494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:56691 (local address 127.0.201.129:35767)
0925 22:45:51.554611 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.554616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.560084 (+ 5468us) server_negotiation.cc:407] Connection header received
0925 22:45:51.560198 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.560200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.560242 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.560304 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:51.560578 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.561064 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.562322 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.562677 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.562759 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.563089 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:51.563151 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:51.563203 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.563322 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.563363 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:45:51.563551 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.553722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:35767 (local address 127.0.201.130:56691)
0925 22:45:51.559939 (+ 6217us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.559955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.560110 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.560336 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.560339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.560346 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:45:51.560471 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.560476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.561222 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.561225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.562182 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.562192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.562806 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.562811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.562905 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.562956 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:45:51.563233 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:45:51.563239 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.563317 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.563394 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6071,"thread_start_us":105,"threads_started":1}
I20260925 22:45:51.585494 2533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:51.585709 2533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:51.588472 2533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5: Sending DeleteTablet for 3 replicas of tablet d10ba0ef41a242538384cb27f29f4062
I20260925 22:45:51.589326 2671 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ba0ef41a242538384cb27f29f4062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:51 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:59944
I20260925 22:45:51.589359 2951 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ba0ef41a242538384cb27f29f4062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:51 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:49460
I20260925 22:45:51.589962 3091 tablet_replica.cc:330] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: stopping tablet replica
I20260925 22:45:51.590318 3091 raft_consensus.cc:2229] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:51.590297 2793 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ba0ef41a242538384cb27f29f4062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:51 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:49958
I20260925 22:45:51.590624 3091 raft_consensus.cc:2258] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:51.591032 2533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:45:51.591454 2533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:51.592434 3091 ts_tablet_manager.cc:1905] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:51.593144 3092 tablet_replica.cc:330] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: stopping tablet replica
I20260925 22:45:51.593427 3092 raft_consensus.cc:2229] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:51.593586 3092 raft_consensus.cc:2258] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:51.593873 3090 tablet_replica.cc:330] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: stopping tablet replica
I20260925 22:45:51.594197 3090 raft_consensus.cc:2229] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:51.594525 3090 raft_consensus.cc:2258] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:51.595773 3092 ts_tablet_manager.cc:1905] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:51.597782 2671 tablet_service.cc:1467] Processing CreateTablet for tablet ca5e0c2ab01840458c3dd8af7b48bd3b (DEFAULT_TABLE table=test-table [id=5702434aec4d40e8a5680d0c81722b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:51.598057 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5e0c2ab01840458c3dd8af7b48bd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:51.598182 2951 tablet_service.cc:1467] Processing CreateTablet for tablet ca5e0c2ab01840458c3dd8af7b48bd3b (DEFAULT_TABLE table=test-table [id=5702434aec4d40e8a5680d0c81722b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:51.598460 2951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5e0c2ab01840458c3dd8af7b48bd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:51.600425 3041 tablet_bootstrap.cc:492] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0: Bootstrap starting.
I20260925 22:45:51.600638 2793 tablet_service.cc:1467] Processing CreateTablet for tablet ca5e0c2ab01840458c3dd8af7b48bd3b (DEFAULT_TABLE table=test-table [id=5702434aec4d40e8a5680d0c81722b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:51.600891 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5e0c2ab01840458c3dd8af7b48bd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:51.601430 3041 tablet_bootstrap.cc:654] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:51.602082 3040 tablet_bootstrap.cc:492] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a: Bootstrap starting.
I20260925 22:45:51.603016 3040 tablet_bootstrap.cc:654] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:51.603307 3042 tablet_bootstrap.cc:492] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb: Bootstrap starting.
I20260925 22:45:51.604246 3042 tablet_bootstrap.cc:654] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:51.606184 3091 ts_tablet_manager.cc:1918] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:45:51.606331 3091 log.cc:1196] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/d10ba0ef41a242538384cb27f29f4062
I20260925 22:45:51.606707 3091 ts_tablet_manager.cc:1939] T d10ba0ef41a242538384cb27f29f4062 P df47a9c9812f41c6b7f04092194db05a: Deleting consensus metadata
I20260925 22:45:51.606796 3090 ts_tablet_manager.cc:1905] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:51.608094 2519 catalog_manager.cc:4841] TS df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365): tablet d10ba0ef41a242538384cb27f29f4062 (table test-table [id=312fe8186a094998855db106fbc45c0d]) successfully deleted
I20260925 22:45:51.610263 3092 ts_tablet_manager.cc:1918] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:45:51.610355 3092 log.cc:1196] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/d10ba0ef41a242538384cb27f29f4062
I20260925 22:45:51.610656 3092 ts_tablet_manager.cc:1939] T d10ba0ef41a242538384cb27f29f4062 P a717d14b129f4dae83fe0984a7ba4cdb: Deleting consensus metadata
I20260925 22:45:51.611837 2519 catalog_manager.cc:4841] TS a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130:45699): tablet d10ba0ef41a242538384cb27f29f4062 (table test-table [id=312fe8186a094998855db106fbc45c0d]) successfully deleted
I20260925 22:45:51.612355 3040 tablet_bootstrap.cc:492] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a: No bootstrap required, opened a new log
I20260925 22:45:51.612444 3040 ts_tablet_manager.cc:1397] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260925 22:45:51.612908 3040 raft_consensus.cc:348] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.613051 3040 raft_consensus.cc:374] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:51.613099 3040 raft_consensus.cc:729] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df47a9c9812f41c6b7f04092194db05a, State: Initialized, Role: FOLLOWER
I20260925 22:45:51.613191 3090 ts_tablet_manager.cc:1918] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:45:51.613282 3090 log.cc:1196] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/d10ba0ef41a242538384cb27f29f4062
I20260925 22:45:51.613233 3040 consensus_queue.cc:260] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.613564 3040 ts_tablet_manager.cc:1428] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:45:51.613584 3090 ts_tablet_manager.cc:1939] T d10ba0ef41a242538384cb27f29f4062 P 1960268384784152be5c4612b4d42fe0: Deleting consensus metadata
I20260925 22:45:51.614636 2518 catalog_manager.cc:4841] TS 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767): tablet d10ba0ef41a242538384cb27f29f4062 (table test-table [id=312fe8186a094998855db106fbc45c0d]) successfully deleted
I20260925 22:45:51.614822 3041 tablet_bootstrap.cc:492] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0: No bootstrap required, opened a new log
I20260925 22:45:51.614899 3041 ts_tablet_manager.cc:1397] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260925 22:45:51.615403 3042 tablet_bootstrap.cc:492] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb: No bootstrap required, opened a new log
I20260925 22:45:51.615353 3041 raft_consensus.cc:348] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.615489 3042 ts_tablet_manager.cc:1397] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20260925 22:45:51.615459 3041 raft_consensus.cc:374] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:51.615532 3041 raft_consensus.cc:729] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1960268384784152be5c4612b4d42fe0, State: Initialized, Role: FOLLOWER
I20260925 22:45:51.615661 3041 consensus_queue.cc:260] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.615935 3041 ts_tablet_manager.cc:1428] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:45:51.615999 3042 raft_consensus.cc:348] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.616115 3042 raft_consensus.cc:374] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:51.616153 3042 raft_consensus.cc:729] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a717d14b129f4dae83fe0984a7ba4cdb, State: Initialized, Role: FOLLOWER
I20260925 22:45:51.616271 3042 consensus_queue.cc:260] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.616544 3042 ts_tablet_manager.cc:1428] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:45:51.637625 3046 raft_consensus.cc:482] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:51.637771 3046 raft_consensus.cc:504] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.638196 3046 leader_election.cc:288] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767), df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365)
I20260925 22:45:51.638666 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5e0c2ab01840458c3dd8af7b48bd3b" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1960268384784152be5c4612b4d42fe0" is_pre_election: true
I20260925 22:45:51.638680 2970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5e0c2ab01840458c3dd8af7b48bd3b" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df47a9c9812f41c6b7f04092194db05a" is_pre_election: true
I20260925 22:45:51.638805 2691 raft_consensus.cc:2454] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 0.
I20260925 22:45:51.638832 2970 raft_consensus.cc:2454] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 0.
I20260925 22:45:51.639115 2762 leader_election.cc:302] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1960268384784152be5c4612b4d42fe0, a717d14b129f4dae83fe0984a7ba4cdb; no voters:
I20260925 22:45:51.639355 3046 raft_consensus.cc:2781] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:51.639408 3046 raft_consensus.cc:482] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:51.639448 3046 raft_consensus.cc:3037] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:51.640751 3046 raft_consensus.cc:504] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.641108 3046 leader_election.cc:288] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 election: Requested vote from peers 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767), df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365)
I20260925 22:45:51.641404 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5e0c2ab01840458c3dd8af7b48bd3b" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1960268384784152be5c4612b4d42fe0"
I20260925 22:45:51.641513 2691 raft_consensus.cc:3037] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:51.641556 2970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5e0c2ab01840458c3dd8af7b48bd3b" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df47a9c9812f41c6b7f04092194db05a"
I20260925 22:45:51.641664 2970 raft_consensus.cc:3037] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:51.642828 2970 raft_consensus.cc:2454] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 1.
I20260925 22:45:51.643050 2691 raft_consensus.cc:2454] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 1.
I20260925 22:45:51.643158 2763 leader_election.cc:302] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a717d14b129f4dae83fe0984a7ba4cdb, df47a9c9812f41c6b7f04092194db05a; no voters:
I20260925 22:45:51.643465 3046 raft_consensus.cc:2781] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:51.643560 3046 raft_consensus.cc:686] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 1 LEADER]: Becoming Leader. State: Replica: a717d14b129f4dae83fe0984a7ba4cdb, State: Running, Role: LEADER
I20260925 22:45:51.643921 3046 consensus_queue.cc:237] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:51.645537 2533 catalog_manager.cc:5495] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb reported cstate change: term changed from 0 to 1, leader changed from <none> to a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130). New cstate: current_term: 1 leader_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260925 22:45:51.663847 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.660504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.0.201.130:45699)
0925 22:45:51.660611 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.660615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.660626 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:45:51.660759 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.660761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.660802 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.660855 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:51.661126 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.661610 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.662776 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.662981 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.663066 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.663368 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:51.663431 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:51.663482 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.663611 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.663652 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:45:51.673317 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.669745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:45699 (local address 127.0.201.130:52327)
0925 22:45:51.669956 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:51.669968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:51.670041 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:51.670288 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:51.670290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:51.670296 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:45:51.670369 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.670373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.671381 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.671387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.672074 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:51.672082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.672628 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.672631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:51.672688 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.672711 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:45:51.672991 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:45:51.672997 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:45:51.673082 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:45:51.673198 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":21}
I20260925 22:45:51.673343 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:51.669843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:52327 (local address 127.0.201.130:45699)
0925 22:45:51.669961 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:51.669972 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:51.670008 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:45:51.670132 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:51.670135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:51.670182 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:51.670260 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:51.670469 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.671239 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.672192 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:51.672520 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:51.672622 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:51.672823 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:51.672897 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:51.672974 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:51.673101 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 22:45:51.673194 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":49}
I20260925 22:45:51.679504 2970 raft_consensus.cc:1261] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Refusing update from remote peer a717d14b129f4dae83fe0984a7ba4cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:51.679849 2691 raft_consensus.cc:1261] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Refusing update from remote peer a717d14b129f4dae83fe0984a7ba4cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:51.680316 3076 consensus_queue.cc:1035] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:51.680467 3064 consensus_queue.cc:1035] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:51.683105 3070 mvcc.cc:205] Tried to move back new op lower bound from 7333381536475074560 to 7333381536334692352. Current Snapshot: MvccSnapshot[applied={T|T < 7333381536475074560}]
I20260925 22:45:51.688619 3071 mvcc.cc:205] Tried to move back new op lower bound from 7333381536475074560 to 7333381536334692352. Current Snapshot: MvccSnapshot[applied={T|T < 7333381536475074560}]
W20260925 22:45:51.696143 3069 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:45:53.252406 2533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:53.252595 2533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:53.254957 2533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5: Sending DeleteTablet for 3 replicas of tablet ca5e0c2ab01840458c3dd8af7b48bd3b
I20260925 22:45:53.255578 2811 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5e0c2ab01840458c3dd8af7b48bd3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:53 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:49958
I20260925 22:45:53.255607 2671 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5e0c2ab01840458c3dd8af7b48bd3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:53 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:59944
I20260925 22:45:53.255627 2951 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5e0c2ab01840458c3dd8af7b48bd3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:53 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:49460
I20260925 22:45:53.255972 3121 tablet_replica.cc:330] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a: stopping tablet replica
I20260925 22:45:53.256000 3120 tablet_replica.cc:330] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb: stopping tablet replica
I20260925 22:45:53.256137 3121 raft_consensus.cc:2229] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:53.256193 3120 raft_consensus.cc:2229] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:53.256251 3121 raft_consensus.cc:2258] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:53.256520 3120 raft_consensus.cc:2258] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:53.256929 3121 ts_tablet_manager.cc:1905] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:53.257299 3120 ts_tablet_manager.cc:1905] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:53.259600 3121 ts_tablet_manager.cc:1918] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:45:53.259701 3121 log.cc:1196] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/ca5e0c2ab01840458c3dd8af7b48bd3b
I20260925 22:45:53.259999 3121 ts_tablet_manager.cc:1939] T ca5e0c2ab01840458c3dd8af7b48bd3b P df47a9c9812f41c6b7f04092194db05a: Deleting consensus metadata
I20260925 22:45:53.260576 3120 ts_tablet_manager.cc:1918] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:45:53.260676 3119 tablet_replica.cc:330] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0: stopping tablet replica
I20260925 22:45:53.260684 3120 log.cc:1196] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/ca5e0c2ab01840458c3dd8af7b48bd3b
I20260925 22:45:53.260665 2519 catalog_manager.cc:4841] TS df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365): tablet ca5e0c2ab01840458c3dd8af7b48bd3b (table test-table [id=5702434aec4d40e8a5680d0c81722b68]) successfully deleted
I20260925 22:45:53.260800 3119 raft_consensus.cc:2229] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:53.260929 3119 raft_consensus.cc:2258] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:53.261034 3120 ts_tablet_manager.cc:1939] T ca5e0c2ab01840458c3dd8af7b48bd3b P a717d14b129f4dae83fe0984a7ba4cdb: Deleting consensus metadata
I20260925 22:45:53.261590 3119 ts_tablet_manager.cc:1905] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:53.261713 2519 catalog_manager.cc:4841] TS a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130:45699): tablet ca5e0c2ab01840458c3dd8af7b48bd3b (table test-table [id=5702434aec4d40e8a5680d0c81722b68]) successfully deleted
I20260925 22:45:53.263968 3119 ts_tablet_manager.cc:1918] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:45:53.264052 3119 log.cc:1196] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/ca5e0c2ab01840458c3dd8af7b48bd3b
I20260925 22:45:53.264302 3119 ts_tablet_manager.cc:1939] T ca5e0c2ab01840458c3dd8af7b48bd3b P 1960268384784152be5c4612b4d42fe0: Deleting consensus metadata
I20260925 22:45:53.265038 2518 catalog_manager.cc:4841] TS 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767): tablet ca5e0c2ab01840458c3dd8af7b48bd3b (table test-table [id=5702434aec4d40e8a5680d0c81722b68]) successfully deleted
Sep 25 22:45:53 dist-test-slave-vdtf krb5kdc[2489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376353, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:45:53 dist-test-slave-vdtf krb5kdc[2489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376353, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:53.285658 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.276239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.0.201.190:44933)
0925 22:45:53.276503 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:53.276515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:53.276617 (+ 102us) server_negotiation.cc:407] Connection header received
0925 22:45:53.276801 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:53.276804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:53.276860 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:53.276956 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:53.277718 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.278286 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.279180 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.279424 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.279557 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.282131 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:53.282134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:53.282136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:53.282159 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:53.283841 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:53.284353 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:53.284359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:53.284361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:53.284464 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:53.284758 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:53.284762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:53.284763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:53.284956 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:53.285116 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:53.285358 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 22:45:53.285482 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260925 22:45:53.288317 2533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:45:53.288707 2533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:53.293365 2671 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed803dce1de4227b4bb0e8b7b2e93c0 (DEFAULT_TABLE table=test-table [id=0820b501d85e49c99835ff842e444f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:53.293540 2951 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed803dce1de4227b4bb0e8b7b2e93c0 (DEFAULT_TABLE table=test-table [id=0820b501d85e49c99835ff842e444f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:53.293701 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed803dce1de4227b4bb0e8b7b2e93c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:53.293829 2951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed803dce1de4227b4bb0e8b7b2e93c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:53.296026 2811 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed803dce1de4227b4bb0e8b7b2e93c0 (DEFAULT_TABLE table=test-table [id=0820b501d85e49c99835ff842e444f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:53.296332 2811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed803dce1de4227b4bb0e8b7b2e93c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:53.296476 3131 tablet_bootstrap.cc:492] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a: Bootstrap starting.
I20260925 22:45:53.297303 3131 tablet_bootstrap.cc:654] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:53.298684 3131 tablet_bootstrap.cc:492] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a: No bootstrap required, opened a new log
I20260925 22:45:53.298779 3131 ts_tablet_manager.cc:1397] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:45:53.298852 3133 tablet_bootstrap.cc:492] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb: Bootstrap starting.
I20260925 22:45:53.298879 3130 tablet_bootstrap.cc:492] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0: Bootstrap starting.
I20260925 22:45:53.299388 3131 raft_consensus.cc:348] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.299512 3131 raft_consensus.cc:374] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:53.299561 3131 raft_consensus.cc:729] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df47a9c9812f41c6b7f04092194db05a, State: Initialized, Role: FOLLOWER
I20260925 22:45:53.299775 3130 tablet_bootstrap.cc:654] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:53.299742 3131 consensus_queue.cc:260] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.299830 3133 tablet_bootstrap.cc:654] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:53.300635 3131 ts_tablet_manager.cc:1428] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:53.300997 3130 tablet_bootstrap.cc:492] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0: No bootstrap required, opened a new log
I20260925 22:45:53.301005 3133 tablet_bootstrap.cc:492] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb: No bootstrap required, opened a new log
I20260925 22:45:53.301086 3133 ts_tablet_manager.cc:1397] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:45:53.301086 3130 ts_tablet_manager.cc:1397] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 22:45:53.301466 3130 raft_consensus.cc:348] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.301466 3133 raft_consensus.cc:348] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.301576 3133 raft_consensus.cc:374] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:53.301576 3130 raft_consensus.cc:374] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:53.301618 3130 raft_consensus.cc:729] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1960268384784152be5c4612b4d42fe0, State: Initialized, Role: FOLLOWER
I20260925 22:45:53.301618 3133 raft_consensus.cc:729] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a717d14b129f4dae83fe0984a7ba4cdb, State: Initialized, Role: FOLLOWER
I20260925 22:45:53.301743 3130 consensus_queue.cc:260] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.301743 3133 consensus_queue.cc:260] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.302013 3133 ts_tablet_manager.cc:1428] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:45:53.302073 3130 ts_tablet_manager.cc:1428] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:45:53.432756 3107 raft_consensus.cc:482] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:53.432899 3107 raft_consensus.cc:504] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.433385 3107 leader_election.cc:288] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767), df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365)
I20260925 22:45:53.433924 2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed803dce1de4227b4bb0e8b7b2e93c0" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1960268384784152be5c4612b4d42fe0" is_pre_election: true
I20260925 22:45:53.433948 2971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed803dce1de4227b4bb0e8b7b2e93c0" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df47a9c9812f41c6b7f04092194db05a" is_pre_election: true
I20260925 22:45:53.434118 2690 raft_consensus.cc:2454] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 0.
I20260925 22:45:53.434130 2971 raft_consensus.cc:2454] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 0.
I20260925 22:45:53.434527 2763 leader_election.cc:302] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1960268384784152be5c4612b4d42fe0, a717d14b129f4dae83fe0984a7ba4cdb, df47a9c9812f41c6b7f04092194db05a; no voters:
I20260925 22:45:53.434719 3107 raft_consensus.cc:2781] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:53.434785 3107 raft_consensus.cc:482] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:53.434826 3107 raft_consensus.cc:3037] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:53.435847 3107 raft_consensus.cc:504] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.436144 3107 leader_election.cc:288] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 election: Requested vote from peers 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767), df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365)
I20260925 22:45:53.436478 2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed803dce1de4227b4bb0e8b7b2e93c0" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1960268384784152be5c4612b4d42fe0"
I20260925 22:45:53.436523 2971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed803dce1de4227b4bb0e8b7b2e93c0" candidate_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df47a9c9812f41c6b7f04092194db05a"
I20260925 22:45:53.436582 2690 raft_consensus.cc:3037] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:53.436650 2971 raft_consensus.cc:3037] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:53.437454 2690 raft_consensus.cc:2454] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 1.
I20260925 22:45:53.437649 2971 raft_consensus.cc:2454] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a717d14b129f4dae83fe0984a7ba4cdb in term 1.
I20260925 22:45:53.437754 2762 leader_election.cc:302] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1960268384784152be5c4612b4d42fe0, a717d14b129f4dae83fe0984a7ba4cdb; no voters:
I20260925 22:45:53.437917 3107 raft_consensus.cc:2781] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:53.438021 3107 raft_consensus.cc:686] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 LEADER]: Becoming Leader. State: Replica: a717d14b129f4dae83fe0984a7ba4cdb, State: Running, Role: LEADER
I20260925 22:45:53.438165 3107 consensus_queue.cc:237] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.440006 2532 catalog_manager.cc:5495] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb reported cstate change: term changed from 0 to 1, leader changed from <none> to a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130). New cstate: current_term: 1 leader_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260925 22:45:53.456296 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.452982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.0.201.130:45699)
0925 22:45:53.453247 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:53.453269 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:53.453284 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:45:53.453317 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:53.453319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:53.453356 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:53.453453 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:53.453718 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.454249 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.455136 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.455381 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.455500 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.455795 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:53.455873 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:53.455962 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:53.456136 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 22:45:53.456185 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
I20260925 22:45:53.462149 2971 raft_consensus.cc:1261] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Refusing update from remote peer a717d14b129f4dae83fe0984a7ba4cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:53.462149 2690 raft_consensus.cc:1261] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Refusing update from remote peer a717d14b129f4dae83fe0984a7ba4cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:53.462560 3107 consensus_queue.cc:1035] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:53.462694 3102 consensus_queue.cc:1035] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:53.492106 2534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:53.492245 2534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:53.494211 2534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5: Sending DeleteTablet for 3 replicas of tablet 5ed803dce1de4227b4bb0e8b7b2e93c0
I20260925 22:45:53.494740 2671 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ed803dce1de4227b4bb0e8b7b2e93c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:53 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:59944
I20260925 22:45:53.494884 2811 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ed803dce1de4227b4bb0e8b7b2e93c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:53 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:49958
I20260925 22:45:53.494925 3119 tablet_replica.cc:330] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0: stopping tablet replica
I20260925 22:45:53.495041 3119 raft_consensus.cc:2229] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:53.495047 3120 tablet_replica.cc:330] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb: stopping tablet replica
I20260925 22:45:53.495151 3119 raft_consensus.cc:2258] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:53.495131 2951 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ed803dce1de4227b4bb0e8b7b2e93c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:53 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:49460
I20260925 22:45:53.495169 3120 raft_consensus.cc:2229] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:53.495324 3121 tablet_replica.cc:330] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a: stopping tablet replica
I20260925 22:45:53.495368 3120 raft_consensus.cc:2258] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:53.495446 3121 raft_consensus.cc:2229] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:53.495597 3121 raft_consensus.cc:2258] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:53.495615 2534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:45:53.496086 3120 ts_tablet_manager.cc:1905] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:45:53.496083 2534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:53.496495 3121 ts_tablet_manager.cc:1905] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:53.498582 3120 ts_tablet_manager.cc:1918] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:45:53.498644 3120 log.cc:1196] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/5ed803dce1de4227b4bb0e8b7b2e93c0
I20260925 22:45:53.498908 3120 ts_tablet_manager.cc:1939] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P a717d14b129f4dae83fe0984a7ba4cdb: Deleting consensus metadata
I20260925 22:45:53.498900 3121 ts_tablet_manager.cc:1918] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:45:53.498971 3121 log.cc:1196] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/5ed803dce1de4227b4bb0e8b7b2e93c0
I20260925 22:45:53.499240 3121 ts_tablet_manager.cc:1939] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P df47a9c9812f41c6b7f04092194db05a: Deleting consensus metadata
I20260925 22:45:53.499801 3119 ts_tablet_manager.cc:1905] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:53.500059 2519 catalog_manager.cc:4841] TS df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365): tablet 5ed803dce1de4227b4bb0e8b7b2e93c0 (table test-table [id=0820b501d85e49c99835ff842e444f2c]) successfully deleted
I20260925 22:45:53.501986 3119 ts_tablet_manager.cc:1918] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:45:53.502058 3119 log.cc:1196] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/5ed803dce1de4227b4bb0e8b7b2e93c0
I20260925 22:45:53.502231 3119 ts_tablet_manager.cc:1939] T 5ed803dce1de4227b4bb0e8b7b2e93c0 P 1960268384784152be5c4612b4d42fe0: Deleting consensus metadata
I20260925 22:45:53.502586 2519 catalog_manager.cc:4841] TS a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130:45699): tablet 5ed803dce1de4227b4bb0e8b7b2e93c0 (table test-table [id=0820b501d85e49c99835ff842e444f2c]) successfully deleted
I20260925 22:45:53.503104 2518 catalog_manager.cc:4841] TS 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767): tablet 5ed803dce1de4227b4bb0e8b7b2e93c0 (table test-table [id=0820b501d85e49c99835ff842e444f2c]) successfully deleted
I20260925 22:45:53.503373 2671 tablet_service.cc:1467] Processing CreateTablet for tablet bbfda40e732e45208bbae2435613e69b (DEFAULT_TABLE table=test-table [id=8ff0445923024c5abe0ce422d94cb0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:53.503624 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbfda40e732e45208bbae2435613e69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:53.503634 2951 tablet_service.cc:1467] Processing CreateTablet for tablet bbfda40e732e45208bbae2435613e69b (DEFAULT_TABLE table=test-table [id=8ff0445923024c5abe0ce422d94cb0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:53.503875 2951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbfda40e732e45208bbae2435613e69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:53.505746 3130 tablet_bootstrap.cc:492] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0: Bootstrap starting.
I20260925 22:45:53.505913 3131 tablet_bootstrap.cc:492] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a: Bootstrap starting.
I20260925 22:45:53.506073 2811 tablet_service.cc:1467] Processing CreateTablet for tablet bbfda40e732e45208bbae2435613e69b (DEFAULT_TABLE table=test-table [id=8ff0445923024c5abe0ce422d94cb0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:53.506268 2811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbfda40e732e45208bbae2435613e69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:53.506660 3130 tablet_bootstrap.cc:654] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:53.506765 3131 tablet_bootstrap.cc:654] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:53.507891 3130 tablet_bootstrap.cc:492] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0: No bootstrap required, opened a new log
I20260925 22:45:53.507915 3131 tablet_bootstrap.cc:492] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a: No bootstrap required, opened a new log
I20260925 22:45:53.507982 3130 ts_tablet_manager.cc:1397] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 22:45:53.507982 3131 ts_tablet_manager.cc:1397] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:45:53.508332 3133 tablet_bootstrap.cc:492] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb: Bootstrap starting.
I20260925 22:45:53.508373 3130 raft_consensus.cc:348] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.508373 3131 raft_consensus.cc:348] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.508490 3130 raft_consensus.cc:374] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:53.508490 3131 raft_consensus.cc:374] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:53.508534 3130 raft_consensus.cc:729] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1960268384784152be5c4612b4d42fe0, State: Initialized, Role: FOLLOWER
I20260925 22:45:53.508539 3131 raft_consensus.cc:729] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df47a9c9812f41c6b7f04092194db05a, State: Initialized, Role: FOLLOWER
I20260925 22:45:53.508661 3131 consensus_queue.cc:260] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.508661 3130 consensus_queue.cc:260] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.508924 3130 ts_tablet_manager.cc:1428] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:45:53.508924 3131 ts_tablet_manager.cc:1428] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:45:53.509153 3133 tablet_bootstrap.cc:654] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:53.510197 3133 tablet_bootstrap.cc:492] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb: No bootstrap required, opened a new log
I20260925 22:45:53.510301 3133 ts_tablet_manager.cc:1397] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:45:53.510702 3133 raft_consensus.cc:348] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.510815 3133 raft_consensus.cc:374] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:53.510856 3133 raft_consensus.cc:729] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a717d14b129f4dae83fe0984a7ba4cdb, State: Initialized, Role: FOLLOWER
I20260925 22:45:53.510984 3133 consensus_queue.cc:260] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.511251 3133 ts_tablet_manager.cc:1428] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:45:53.612255 3134 raft_consensus.cc:482] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:53.612396 3134 raft_consensus.cc:504] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.613536 3134 leader_election.cc:288] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767), a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130:45699)
I20260925 22:45:53.617480 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.613928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:40055 (local address 127.0.201.129:35767)
0925 22:45:53.614155 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:53.614172 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:53.614265 (+ 93us) server_negotiation.cc:407] Connection header received
0925 22:45:53.614428 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:53.614431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:53.614477 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:53.614581 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:53.614832 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.615361 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.616500 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.616907 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.617041 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.617097 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:53.617277 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 22:45:53.617345 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":54,"threads_started":1}
I20260925 22:45:53.617487 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.613815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:35767 (local address 127.0.201.131:40055)
0925 22:45:53.614158 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:53.614173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:53.614323 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:53.614592 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:53.614595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:53.614609 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:53.614726 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:53.614731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.615484 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.615488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:53.616374 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:53.616383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.617041 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.617044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:53.617153 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.617172 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:45:53.617263 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:45:53.617335 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":80,"threads_started":1}
I20260925 22:45:53.617609 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.614012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:38071 (local address 127.0.201.130:45699)
0925 22:45:53.614157 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:53.614161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:53.614424 (+ 263us) server_negotiation.cc:407] Connection header received
0925 22:45:53.614583 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:53.614586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:53.614634 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:53.614723 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:53.614983 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.615477 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.616597 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.617008 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.617159 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.617212 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:53.617417 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 22:45:53.617486 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:45:53.617650 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.613815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:45699 (local address 127.0.201.131:38071)
0925 22:45:53.614322 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:53.614334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:53.614472 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:53.614730 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:53.614733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:53.614748 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:53.614866 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:53.614871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.615611 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.615614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:53.616465 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:53.616474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.617154 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.617158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:53.617268 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.617288 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:45:53.617399 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:45:53.617474 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":179,"thread_start_us":112,"threads_started":1}
I20260925 22:45:53.617916 2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbfda40e732e45208bbae2435613e69b" candidate_uuid: "df47a9c9812f41c6b7f04092194db05a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1960268384784152be5c4612b4d42fe0" is_pre_election: true
I20260925 22:45:53.618024 2831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbfda40e732e45208bbae2435613e69b" candidate_uuid: "df47a9c9812f41c6b7f04092194db05a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" is_pre_election: true
I20260925 22:45:53.618064 2690 raft_consensus.cc:2454] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df47a9c9812f41c6b7f04092194db05a in term 0.
I20260925 22:45:53.618145 2831 raft_consensus.cc:2454] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df47a9c9812f41c6b7f04092194db05a in term 0.
I20260925 22:45:53.618439 2902 leader_election.cc:302] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1960268384784152be5c4612b4d42fe0, a717d14b129f4dae83fe0984a7ba4cdb, df47a9c9812f41c6b7f04092194db05a; no voters:
I20260925 22:45:53.618745 3134 raft_consensus.cc:2781] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:53.618824 3134 raft_consensus.cc:482] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:53.618877 3134 raft_consensus.cc:3037] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:53.620302 3134 raft_consensus.cc:504] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.620656 3134 leader_election.cc:288] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [CANDIDATE]: Term 1 election: Requested vote from peers 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767), a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130:45699)
I20260925 22:45:53.621019 2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbfda40e732e45208bbae2435613e69b" candidate_uuid: "df47a9c9812f41c6b7f04092194db05a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1960268384784152be5c4612b4d42fe0"
I20260925 22:45:53.621071 2831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbfda40e732e45208bbae2435613e69b" candidate_uuid: "df47a9c9812f41c6b7f04092194db05a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a717d14b129f4dae83fe0984a7ba4cdb"
I20260925 22:45:53.621145 2690 raft_consensus.cc:3037] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:53.621213 2831 raft_consensus.cc:3037] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:53.622063 2690 raft_consensus.cc:2454] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df47a9c9812f41c6b7f04092194db05a in term 1.
I20260925 22:45:53.622069 2831 raft_consensus.cc:2454] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df47a9c9812f41c6b7f04092194db05a in term 1.
I20260925 22:45:53.622372 2902 leader_election.cc:302] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1960268384784152be5c4612b4d42fe0, df47a9c9812f41c6b7f04092194db05a; no voters:
I20260925 22:45:53.622651 3134 raft_consensus.cc:2781] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:53.622885 3134 raft_consensus.cc:686] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 1 LEADER]: Becoming Leader. State: Replica: df47a9c9812f41c6b7f04092194db05a, State: Running, Role: LEADER
I20260925 22:45:53.623149 3134 consensus_queue.cc:237] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } }
I20260925 22:45:53.625821 2534 catalog_manager.cc:5495] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a reported cstate change: term changed from 0 to 1, leader changed from <none> to df47a9c9812f41c6b7f04092194db05a (127.0.201.131). New cstate: current_term: 1 leader_uuid: "df47a9c9812f41c6b7f04092194db05a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df47a9c9812f41c6b7f04092194db05a" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 46365 } health_report { overall_health: HEALTHY } } }
I20260925 22:45:53.667428 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.664014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.0.201.131:46365)
0925 22:45:53.664281 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:53.664300 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:53.664312 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:45:53.664396 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:53.664398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:53.664453 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:53.664568 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:53.664819 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.665360 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.666132 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.666352 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.666427 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.666640 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:53.667083 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:53.667186 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:53.667261 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 22:45:53.667306 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":119,"threads_started":1}
I20260925 22:45:53.677930 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.674318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:58453 (local address 127.0.201.130:45699)
0925 22:45:53.674456 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:53.674461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:53.674670 (+ 209us) server_negotiation.cc:407] Connection header received
0925 22:45:53.674837 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:53.674839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:53.674893 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:53.674971 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:53.675216 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.675933 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.676809 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.677032 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.677159 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.677444 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:53.677535 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:53.677646 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:53.677740 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 22:45:53.677788 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:45:53.677959 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.674191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:45699 (local address 127.0.201.131:58453)
0925 22:45:53.674581 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:53.674594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:53.674711 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:53.674989 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:53.674992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:53.674998 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:45:53.675104 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:53.675110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.676046 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.676063 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:53.676684 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:53.676693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.677170 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.677174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:53.677261 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.677308 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:45:53.677655 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:45:53.677664 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:45:53.677731 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 22:45:53.677808 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":94,"threads_started":1}
I20260925 22:45:53.683400 2690 raft_consensus.cc:1261] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Refusing update from remote peer df47a9c9812f41c6b7f04092194db05a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:53.683400 2831 raft_consensus.cc:1261] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Refusing update from remote peer df47a9c9812f41c6b7f04092194db05a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:53.683918 3151 consensus_queue.cc:1035] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a717d14b129f4dae83fe0984a7ba4cdb" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:53.684230 3134 consensus_queue.cc:1035] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1960268384784152be5c4612b4d42fe0" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:53.700588 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.696946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:33529 (local address 127.0.201.131:46365)
0925 22:45:53.697066 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:53.697071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:53.697258 (+ 187us) server_negotiation.cc:407] Connection header received
0925 22:45:53.697383 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:53.697386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:53.697425 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:53.697519 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:53.697764 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.698272 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.699366 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.699755 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.699856 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.700121 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:53.700195 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:53.700271 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:53.700386 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 22:45:53.700473 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:45:53.700603 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:53.696845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:46365 (local address 127.0.201.130:33529)
0925 22:45:53.697168 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:53.697180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:53.697298 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:53.697527 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:53.697530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:53.697537 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:45:53.697646 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:53.697655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.698388 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.698391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:53.699250 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:53.699260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:53.699854 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:53.699857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:53.699938 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:53.699993 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:45:53.700296 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:45:53.700302 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:45:53.700374 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:45:53.700454 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
W20260925 22:45:53.701442 3103 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:45:55.256425 2534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:55.256579 2534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:55.259241 2534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 836ee11b65944d13aaba8c92f93e25e5: Sending DeleteTablet for 3 replicas of tablet bbfda40e732e45208bbae2435613e69b
I20260925 22:45:55.259862 2671 tablet_service.cc:1514] Processing DeleteTablet for tablet bbfda40e732e45208bbae2435613e69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:55 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:59944
I20260925 22:45:55.259934 2951 tablet_service.cc:1514] Processing DeleteTablet for tablet bbfda40e732e45208bbae2435613e69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:55 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:49460
I20260925 22:45:55.260004 2811 tablet_service.cc:1514] Processing DeleteTablet for tablet bbfda40e732e45208bbae2435613e69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:55 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:49958
I20260925 22:45:55.260264 3185 tablet_replica.cc:330] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0: stopping tablet replica
I20260925 22:45:55.260324 3187 tablet_replica.cc:330] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb: stopping tablet replica
I20260925 22:45:55.260411 3185 raft_consensus.cc:2229] T bbfda40e732e45208bbae2435613e69b P 1960268384784152be5c4612b4d42fe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:55.260450 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 2606
I20260925 22:45:55.260453 3187 raft_consensus.cc:2229] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:55.260453 3186 tablet_replica.cc:330] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a: stopping tablet replica
I20260925 22:45:55.261344 3187 raft_consensus.cc:2258] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:55.261373 3186 raft_consensus.cc:2229] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:55.261868 3186 raft_consensus.cc:2258] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:55.262084 3187 ts_tablet_manager.cc:1905] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:55.262619 3186 ts_tablet_manager.cc:1905] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:55.264372 3187 ts_tablet_manager.cc:1918] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:45:55.264470 3187 log.cc:1196] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/bbfda40e732e45208bbae2435613e69b
I20260925 22:45:55.264719 3187 ts_tablet_manager.cc:1939] T bbfda40e732e45208bbae2435613e69b P a717d14b129f4dae83fe0984a7ba4cdb: Deleting consensus metadata
I20260925 22:45:55.265544 2519 catalog_manager.cc:4841] TS a717d14b129f4dae83fe0984a7ba4cdb (127.0.201.130:45699): tablet bbfda40e732e45208bbae2435613e69b (table test-table [id=8ff0445923024c5abe0ce422d94cb0a5]) successfully deleted
I20260925 22:45:55.265971 3186 ts_tablet_manager.cc:1918] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:45:55.266053 3186 log.cc:1196] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/bbfda40e732e45208bbae2435613e69b
I20260925 22:45:55.266335 3186 ts_tablet_manager.cc:1939] T bbfda40e732e45208bbae2435613e69b P df47a9c9812f41c6b7f04092194db05a: Deleting consensus metadata
I20260925 22:45:55.267136 2519 catalog_manager.cc:4841] TS df47a9c9812f41c6b7f04092194db05a (127.0.201.131:46365): tablet bbfda40e732e45208bbae2435613e69b (table test-table [id=8ff0445923024c5abe0ce422d94cb0a5]) successfully deleted
W20260925 22:45:55.269613 2518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:35767 (error 108)
I20260925 22:45:55.269855 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 2746
I20260925 22:45:55.271265 3189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:55.270882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:35767 (local address 127.0.0.1:43690)
0925 22:45:55.271124 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:55.271187 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
W20260925 22:45:55.271392 2518 catalog_manager.cc:4584] TS 1960268384784152be5c4612b4d42fe0 (127.0.201.129:35767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbfda40e732e45208bbae2435613e69b: Network error: Client connection negotiation failed: client connection to 127.0.201.129:35767: connect: Connection refused (error 111)
I20260925 22:45:55.277799 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 2886
I20260925 22:45:55.284281 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 2502
2026-09-25T22:45:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6413 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:55 dist-test-slave-vdtf krb5kdc[3192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:55 dist-test-slave-vdtf krb5kdc[3192](info): set up 2 sockets
Sep 25 22:45:55 dist-test-slave-vdtf krb5kdc[3192](info): commencing operation
krb5kdc: starting...
W20260925 22:45:56.339572 806 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:45:56 dist-test-slave-vdtf krb5kdc[3192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:45:56.458904 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37779
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:37779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:56.565830 3205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:56.566140 3205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:56.566218 3205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:56.570166 3205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:56.570278 3205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:56.570315 3205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:56.570393 3205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:56.570429 3205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:56.575798 3205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:37779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:56.576864 3205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:56.577889 3205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:56.586898 3213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:56.586958 3210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:56.586943 3211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:56.586915 3205 server_base.cc:1034] running on GCE node
I20260925 22:45:56.587932 3205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:56.589033 3205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:56.590257 3205 hybrid_clock.cc:648] HybridClock initialized: now 1790376356590226 us; error 43 us; skew 500 ppm
Sep 25 22:45:56 dist-test-slave-vdtf krb5kdc[3192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:56.593045 3205 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:45:56.593859 3205 webserver.cc:460] Webserver started at http://127.0.201.190:43073/ using document root <none> and password file <none>
I20260925 22:45:56.594374 3205 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:56.594429 3205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:56.594589 3205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:56.596613 3205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "4142e4fddfc247ffbfd13b7a64390658"
format_stamp: "Formatted at 2026-09-25 22:45:56 on dist-test-slave-vdtf"
server_key: "331506f461cd2ca75c360ab86ca01029"
server_key_iv: "37d4146b07613613a40b0e226c63cea9"
server_key_version: "encryptionkey@0"
I20260925 22:45:56.597128 3205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "4142e4fddfc247ffbfd13b7a64390658"
format_stamp: "Formatted at 2026-09-25 22:45:56 on dist-test-slave-vdtf"
server_key: "331506f461cd2ca75c360ab86ca01029"
server_key_iv: "37d4146b07613613a40b0e226c63cea9"
server_key_version: "encryptionkey@0"
I20260925 22:45:56.600672 3205 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:45:56.602953 3220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:56.604198 3205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:45:56.604323 3205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "4142e4fddfc247ffbfd13b7a64390658"
format_stamp: "Formatted at 2026-09-25 22:45:56 on dist-test-slave-vdtf"
server_key: "331506f461cd2ca75c360ab86ca01029"
server_key_iv: "37d4146b07613613a40b0e226c63cea9"
server_key_version: "encryptionkey@0"
I20260925 22:45:56.604421 3205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:56.613970 3205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:56.616842 3205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:56.617028 3205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:56.623970 3205 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:37779
I20260925 22:45:56.623991 3272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:37779 every 8 connection(s)
I20260925 22:45:56.624915 3205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:56.627601 3273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:56.632827 3273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658: Bootstrap starting.
I20260925 22:45:56.634454 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 3205
I20260925 22:45:56.634554 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:45:56.634825 806 external_mini_cluster.cc:1428] Setting key 193f2cde4be7068d761c2092468a3a03
I20260925 22:45:56.635546 3273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:56.636431 3273 log.cc:824] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:56.638538 3273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658: No bootstrap required, opened a new log
I20260925 22:45:56.642750 3273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4142e4fddfc247ffbfd13b7a64390658" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37779 } }
I20260925 22:45:56.643031 3273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:56.643124 3273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4142e4fddfc247ffbfd13b7a64390658, State: Initialized, Role: FOLLOWER
I20260925 22:45:56.643549 3273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4142e4fddfc247ffbfd13b7a64390658" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37779 } }
Sep 25 22:45:56 dist-test-slave-vdtf krb5kdc[3192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:56.643663 3273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:56.643785 3273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:56.643877 3273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:56.644770 3273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4142e4fddfc247ffbfd13b7a64390658" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37779 } }
I20260925 22:45:56.645107 3273 leader_election.cc:302] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4142e4fddfc247ffbfd13b7a64390658; no voters:
I20260925 22:45:56.645522 3273 leader_election.cc:288] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:56.645587 3278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:56.645790 3278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [term 1 LEADER]: Becoming Leader. State: Replica: 4142e4fddfc247ffbfd13b7a64390658, State: Running, Role: LEADER
I20260925 22:45:56.646129 3278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4142e4fddfc247ffbfd13b7a64390658" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37779 } }
I20260925 22:45:56.646763 3273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:56.648494 3279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4142e4fddfc247ffbfd13b7a64390658. Latest consensus state: current_term: 1 leader_uuid: "4142e4fddfc247ffbfd13b7a64390658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4142e4fddfc247ffbfd13b7a64390658" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37779 } } }
I20260925 22:45:56.648615 3279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:56.648989 3292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:56.649255 3280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4142e4fddfc247ffbfd13b7a64390658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4142e4fddfc247ffbfd13b7a64390658" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37779 } } }
I20260925 22:45:56.649540 3280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:56.649993 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:56.635917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.0.201.190:37779)
0925 22:45:56.636344 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:56.636359 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:56.636394 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:45:56.637019 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:56.637035 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:56.637371 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:56.637664 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:56.638511 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:56.639502 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:56.640729 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:56.641185 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:56.641300 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:56.644306 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:56.644310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:56.644323 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:56.644347 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:56.646271 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:56.646711 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:56.646715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:56.646720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:56.646821 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:56.647233 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:56.647238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:56.647241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:56.647750 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:56.647937 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:56.648387 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 22:45:56.648690 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":164,"threads_started":1}
I20260925 22:45:56.652364 3292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:56.657202 3292 catalog_manager.cc:1347] Generated new cluster ID: 1a8dc3faed384535b542eb1e2ca420ed
I20260925 22:45:56.657264 3292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:56.671938 3292 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:56.672832 3292 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:56.680109 3292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4142e4fddfc247ffbfd13b7a64390658: Generated new TSK 0
I20260925 22:45:56.680760 3292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:45:56.719957 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37779
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:56.856317 3299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:56.856571 3299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:56.856613 3299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:56.861241 3299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:56.861303 3299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:56.861402 3299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:45:56.866874 3299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:56.867985 3299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:56.869043 3299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:56.880184 3304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:56.880173 3305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:56.880327 3299 server_base.cc:1034] running on GCE node
W20260925 22:45:56.880175 3307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:56.880856 3299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:56.881505 3299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:56.882669 3299 hybrid_clock.cc:648] HybridClock initialized: now 1790376356882656 us; error 23 us; skew 500 ppm
Sep 25 22:45:56 dist-test-slave-vdtf krb5kdc[3192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:56.885453 3299 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:45:56.886433 3299 webserver.cc:460] Webserver started at http://127.0.201.129:41657/ using document root <none> and password file <none>
I20260925 22:45:56.886998 3299 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:56.887099 3299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:56.887336 3299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:56.889387 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "7e6f29ef50274f928068916b1cb3156b"
format_stamp: "Formatted at 2026-09-25 22:45:56 on dist-test-slave-vdtf"
server_key: "8d5ca64d7cabeb55dc136d095671066f"
server_key_iv: "05071784fa98dbdaffce0e1d3b2012a9"
server_key_version: "encryptionkey@0"
I20260925 22:45:56.889948 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "7e6f29ef50274f928068916b1cb3156b"
format_stamp: "Formatted at 2026-09-25 22:45:56 on dist-test-slave-vdtf"
server_key: "8d5ca64d7cabeb55dc136d095671066f"
server_key_iv: "05071784fa98dbdaffce0e1d3b2012a9"
server_key_version: "encryptionkey@0"
I20260925 22:45:56.893705 3299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:45:56.896186 3314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:56.897353 3299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:45:56.897490 3299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "7e6f29ef50274f928068916b1cb3156b"
format_stamp: "Formatted at 2026-09-25 22:45:56 on dist-test-slave-vdtf"
server_key: "8d5ca64d7cabeb55dc136d095671066f"
server_key_iv: "05071784fa98dbdaffce0e1d3b2012a9"
server_key_version: "encryptionkey@0"
I20260925 22:45:56.897605 3299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:56.913266 3299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:56.916165 3299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:56.916371 3299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:56.916950 3299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:56.917749 3299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:56.917824 3299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:56.917891 3299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:56.917941 3299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:56.927090 3299 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:33045
I20260925 22:45:56.927117 3427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:33045 every 8 connection(s)
I20260925 22:45:56.928010 3299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:56.937008 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 3299
I20260925 22:45:56.937134 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:56.937446 806 external_mini_cluster.cc:1428] Setting key a7768c675681c17ff63947237c5b2c45
Sep 25 22:45:56 dist-test-slave-vdtf krb5kdc[3192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:56.942239 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:56.929859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:57763 (local address 127.0.201.190:37779)
0925 22:45:56.929988 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:56.929991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:56.930744 (+ 753us) server_negotiation.cc:407] Connection header received
0925 22:45:56.932273 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:56.932277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:56.932333 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:56.932410 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:56.933519 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:56.934055 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:56.935293 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:56.935541 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:56.935666 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:56.938435 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:56.938438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:56.938441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:56.938492 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:56.940222 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:56.940748 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:56.940754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:56.940755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:56.940834 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:56.941171 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:56.941176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:56.941177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:56.941348 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:56.941459 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:56.941923 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 22:45:56.942038 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:45:56.943010 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:56.930178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37779 (local address 127.0.201.129:57763)
0925 22:45:56.930596 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:56.930637 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:56.932056 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:56.932569 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:56.932575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:56.932911 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:56.933370 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:56.933383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:56.934190 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:56.934194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:56.935168 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:56.935178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:56.935721 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:56.935725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:56.935894 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:56.936511 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:56.936526 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:56.938284 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:56.940336 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:56.940340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:56.940350 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:56.940624 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:56.940938 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:56.940946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:56.940947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:56.941074 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:56.941455 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:56.941461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:56.941696 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 22:45:56.941882 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 22:45:56.942182 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
I20260925 22:45:56.944245 3428 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37779
I20260925 22:45:56.944494 3428 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:56.944962 3428 heartbeater.cc:511] Master 127.0.201.190:37779 requested a full tablet report, sending...
I20260925 22:45:56.946648 3236 ts_manager.cc:194] Registered new tserver with Master: 7e6f29ef50274f928068916b1cb3156b (127.0.201.129:33045)
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:45:56.948092 3236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:57763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:45:56.965111 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37779
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:57.073282 3433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:57.073587 3433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:57.073669 3433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:57.077735 3433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:57.077834 3433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:57.077929 3433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:45:57.083199 3433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:57.084373 3433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:57.085409 3433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:57.095086 3439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:57.095151 3441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:57.095147 3438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:57.095321 3433 server_base.cc:1034] running on GCE node
I20260925 22:45:57.095858 3433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:57.096494 3433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:57.097690 3433 hybrid_clock.cc:648] HybridClock initialized: now 1790376357097679 us; error 31 us; skew 500 ppm
Sep 25 22:45:57 dist-test-slave-vdtf krb5kdc[3192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:57.100531 3433 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:45:57.101382 3433 webserver.cc:460] Webserver started at http://127.0.201.130:37839/ using document root <none> and password file <none>
I20260925 22:45:57.101950 3433 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:57.102010 3433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:57.102181 3433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:57.104192 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "5eff7dd5a3104f4a9818a62f14fe2a07"
format_stamp: "Formatted at 2026-09-25 22:45:57 on dist-test-slave-vdtf"
server_key: "ef7e809e77c5b2f508eca80d0f2173be"
server_key_iv: "4b02c4dfa07dd2009db5e4c2ea310410"
server_key_version: "encryptionkey@0"
I20260925 22:45:57.104693 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "5eff7dd5a3104f4a9818a62f14fe2a07"
format_stamp: "Formatted at 2026-09-25 22:45:57 on dist-test-slave-vdtf"
server_key: "ef7e809e77c5b2f508eca80d0f2173be"
server_key_iv: "4b02c4dfa07dd2009db5e4c2ea310410"
server_key_version: "encryptionkey@0"
I20260925 22:45:57.108376 3433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:57.110733 3448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:57.112181 3433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:57.112308 3433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "5eff7dd5a3104f4a9818a62f14fe2a07"
format_stamp: "Formatted at 2026-09-25 22:45:57 on dist-test-slave-vdtf"
server_key: "ef7e809e77c5b2f508eca80d0f2173be"
server_key_iv: "4b02c4dfa07dd2009db5e4c2ea310410"
server_key_version: "encryptionkey@0"
I20260925 22:45:57.112411 3433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:57.138895 3433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:57.142040 3433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:57.142259 3433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:57.142865 3433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:57.143741 3433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:57.143790 3433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:57.143854 3433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:57.143893 3433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:57.153805 3433 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:36871
I20260925 22:45:57.153825 3561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:36871 every 8 connection(s)
I20260925 22:45:57.154677 3433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:57.160792 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 3433
I20260925 22:45:57.160902 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:57.161233 806 external_mini_cluster.cc:1428] Setting key c554aab45def98df22c68227250b5994
Sep 25 22:45:57 dist-test-slave-vdtf krb5kdc[3192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:57.169261 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:57.156545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:43169 (local address 127.0.201.190:37779)
0925 22:45:57.156684 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:57.156687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:57.157490 (+ 803us) server_negotiation.cc:407] Connection header received
0925 22:45:57.158887 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:57.158891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:57.158947 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:57.159053 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:57.160280 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:57.160795 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:57.162199 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:57.162488 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:57.162578 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:57.165243 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:57.165250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:57.165256 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:57.165288 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:57.167003 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:57.167526 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:57.167533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:57.167537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:57.167635 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:57.168037 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:57.168042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:57.168045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:57.168326 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:57.168444 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:57.168909 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 22:45:57.169048 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:45:57.169966 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:57.156925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37779 (local address 127.0.201.130:43169)
0925 22:45:57.157346 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:57.157387 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:57.158670 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:57.159230 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:57.159236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:57.159575 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:57.160134 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:57.160148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:57.160923 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:57.160927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:57.162083 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:57.162093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:57.162664 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:57.162671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:57.162812 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:57.163418 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:57.163434 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:57.165085 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:57.167128 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:57.167132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:57.167144 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:57.167417 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:57.167797 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:57.167800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:57.167801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:57.167936 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:57.168435 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:57.168440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:57.168662 (+ 222us) client_negotiation.cc:769] Sending connection context
0925 22:45:57.168819 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 22:45:57.169060 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:45:57.171212 3562 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37779
I20260925 22:45:57.171458 3562 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:57.172052 3562 heartbeater.cc:511] Master 127.0.201.190:37779 requested a full tablet report, sending...
I20260925 22:45:57.173108 3236 ts_manager.cc:194] Registered new tserver with Master: 5eff7dd5a3104f4a9818a62f14fe2a07 (127.0.201.130:36871)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
I20260925 22:45:57.173679 3236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:43169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:45:57.191035 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37779
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:57.300547 3567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:57.300891 3567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:57.300972 3567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:57.305112 3567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:57.305193 3567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:57.305279 3567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:45:57.310676 3567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:57.311810 3567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:57.312839 3567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:57.322592 3573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:57.322696 3567 server_base.cc:1034] running on GCE node
W20260925 22:45:57.322608 3575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:57.322592 3572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:57.323269 3567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:57.323942 3567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:57.325134 3567 hybrid_clock.cc:648] HybridClock initialized: now 1790376357325116 us; error 27 us; skew 500 ppm
Sep 25 22:45:57 dist-test-slave-vdtf krb5kdc[3192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:57.327929 3567 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:45:57.328949 3567 webserver.cc:460] Webserver started at http://127.0.201.131:39831/ using document root <none> and password file <none>
I20260925 22:45:57.329522 3567 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:57.329619 3567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:57.329861 3567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:57.331913 3567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "d28d2dc5f6604a9487cd29607733819b"
format_stamp: "Formatted at 2026-09-25 22:45:57 on dist-test-slave-vdtf"
server_key: "c4efdafceb842f33ae9273a94ddb865b"
server_key_iv: "b1a720b448603b437f08daf8a9e6e7f9"
server_key_version: "encryptionkey@0"
I20260925 22:45:57.332463 3567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "d28d2dc5f6604a9487cd29607733819b"
format_stamp: "Formatted at 2026-09-25 22:45:57 on dist-test-slave-vdtf"
server_key: "c4efdafceb842f33ae9273a94ddb865b"
server_key_iv: "b1a720b448603b437f08daf8a9e6e7f9"
server_key_version: "encryptionkey@0"
I20260925 22:45:57.336051 3567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:45:57.338340 3582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:57.339740 3567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:57.339893 3567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "d28d2dc5f6604a9487cd29607733819b"
format_stamp: "Formatted at 2026-09-25 22:45:57 on dist-test-slave-vdtf"
server_key: "c4efdafceb842f33ae9273a94ddb865b"
server_key_iv: "b1a720b448603b437f08daf8a9e6e7f9"
server_key_version: "encryptionkey@0"
I20260925 22:45:57.340005 3567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:57.355033 3567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:57.357828 3567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:57.358036 3567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:57.358608 3567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:57.359409 3567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:57.359483 3567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:57.359551 3567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:57.359607 3567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:57.368696 3567 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:41077
I20260925 22:45:57.368745 3695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:41077 every 8 connection(s)
I20260925 22:45:57.369566 3567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:57.376843 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 3567
I20260925 22:45:57.376971 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:57.377285 806 external_mini_cluster.cc:1428] Setting key eec5f0d6c1ae051984b8598367f1ac71
Sep 25 22:45:57 dist-test-slave-vdtf krb5kdc[3192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:57.382845 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:57.371312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:50269 (local address 127.0.201.190:37779)
0925 22:45:57.371481 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:57.371487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:57.372380 (+ 893us) server_negotiation.cc:407] Connection header received
0925 22:45:57.373714 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:57.373717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:57.373761 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:57.373834 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:57.374946 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:57.375462 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:57.376689 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:57.376895 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:57.377013 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:57.379841 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:57.379844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:57.379846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:57.379870 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:57.381127 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:57.381585 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:57.381588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:57.381589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:57.381664 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:57.381969 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:57.381971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:57.381972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:57.382136 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:57.382212 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:57.382633 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 22:45:57.382735 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:45:57.383752 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:57.371741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37779 (local address 127.0.201.131:50269)
0925 22:45:57.372206 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:57.372258 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:57.373504 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:57.374000 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:57.374007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:57.374355 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:57.374822 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:57.374835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:57.375576 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:57.375579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:57.376532 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:57.376542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:57.377050 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:57.377054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:57.377232 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:57.377840 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:57.377865 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:57.379625 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:57.381230 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:57.381235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:57.381246 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:57.381484 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:57.381757 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:57.381759 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:57.381761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:57.381880 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:57.382218 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:57.382222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:57.382430 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 22:45:57.382572 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 22:45:57.382804 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260925 22:45:57.384824 3696 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37779
I20260925 22:45:57.385066 3696 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:57.385527 3696 heartbeater.cc:511] Master 127.0.201.190:37779 requested a full tablet report, sending...
I20260925 22:45:57.386444 3236 ts_manager.cc:194] Registered new tserver with Master: d28d2dc5f6604a9487cd29607733819b (127.0.201.131:41077)
I20260925 22:45:57.386950 3236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:50269
I20260925 22:45:57.390978 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:45:57.403529 3276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:57.402044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.0.201.190:37779)
0925 22:45:57.402200 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:57.402205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:57.402414 (+ 209us) server_negotiation.cc:407] Connection header received
0925 22:45:57.402580 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:57.402585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:57.402629 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:57.402736 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:57.403432 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42348: BlockingRecv error: recv got EOF from 127.0.0.1:42348 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:45:57.404592 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 3299
I20260925 22:45:57.410036 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 3433
I20260925 22:45:57.415194 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 3567
I20260925 22:45:57.420434 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 3205
2026-09-25T22:45:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2135 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:57 dist-test-slave-vdtf krb5kdc[3708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:57 dist-test-slave-vdtf krb5kdc[3708](info): set up 2 sockets
Sep 25 22:45:57 dist-test-slave-vdtf krb5kdc[3708](info): commencing operation
krb5kdc: starting...
W20260925 22:45:58.463622 806 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:45:58 dist-test-slave-vdtf krb5kdc[3708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:45:58.592839 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:39769
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:39769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:58.703622 3721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:58.703951 3721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:58.704027 3721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:58.707988 3721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:58.708037 3721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:58.708056 3721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:58.708073 3721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:58.708141 3721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:58.713186 3721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:39769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:39769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:45:58.714164 3721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:58.715127 3721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:58.723295 3727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:58.723343 3726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:58.723449 3721 server_base.cc:1034] running on GCE node
W20260925 22:45:58.723310 3729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:58.724138 3721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:58.725207 3721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:58.726366 3721 hybrid_clock.cc:648] HybridClock initialized: now 1790376358726337 us; error 43 us; skew 500 ppm
Sep 25 22:45:58 dist-test-slave-vdtf krb5kdc[3708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:58.728979 3721 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:45:58.729923 3721 webserver.cc:460] Webserver started at http://127.0.201.190:45993/ using document root <none> and password file <none>
I20260925 22:45:58.730468 3721 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:58.730561 3721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:58.730793 3721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:58.732802 3721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec"
format_stamp: "Formatted at 2026-09-25 22:45:58 on dist-test-slave-vdtf"
server_key: "ec42b4c00b8ad20d9594e961148ade73"
server_key_iv: "cae786ff96854515b63b6aea529999e4"
server_key_version: "encryptionkey@0"
I20260925 22:45:58.733357 3721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec"
format_stamp: "Formatted at 2026-09-25 22:45:58 on dist-test-slave-vdtf"
server_key: "ec42b4c00b8ad20d9594e961148ade73"
server_key_iv: "cae786ff96854515b63b6aea529999e4"
server_key_version: "encryptionkey@0"
I20260925 22:45:58.736943 3721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:45:58.739343 3736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:58.740607 3721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:45:58.740787 3721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec"
format_stamp: "Formatted at 2026-09-25 22:45:58 on dist-test-slave-vdtf"
server_key: "ec42b4c00b8ad20d9594e961148ade73"
server_key_iv: "cae786ff96854515b63b6aea529999e4"
server_key_version: "encryptionkey@0"
I20260925 22:45:58.740899 3721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:58.752506 3721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:58.755195 3721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:58.755399 3721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:58.762249 3721 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:39769
I20260925 22:45:58.762526 3788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:39769 every 8 connection(s)
I20260925 22:45:58.763222 3721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:58.765935 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:58.768972 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 3721
I20260925 22:45:58.769095 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:45:58.769381 806 external_mini_cluster.cc:1428] Setting key c6689eea21a0f827bfbec34b3ea0f459
I20260925 22:45:58.771886 3789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec: Bootstrap starting.
I20260925 22:45:58.774346 3789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:58.775095 3789 log.cc:824] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:58.776968 3789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec: No bootstrap required, opened a new log
Sep 25 22:45:58 dist-test-slave-vdtf krb5kdc[3708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:45:58.780608 3789 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 39769 } }
I20260925 22:45:58.780821 3789 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:58.780896 3789 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7162e6c7de347ff81a8a8b4fc0d35ec, State: Initialized, Role: FOLLOWER
I20260925 22:45:58.781363 3789 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 39769 } }
I20260925 22:45:58.781482 3789 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:58.781564 3789 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:58.781657 3789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:58.782548 3789 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 39769 } }
I20260925 22:45:58.782896 3789 leader_election.cc:302] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7162e6c7de347ff81a8a8b4fc0d35ec; no voters:
I20260925 22:45:58.783218 3789 leader_election.cc:288] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:58.783329 3794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:58.783511 3794 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [term 1 LEADER]: Becoming Leader. State: Replica: a7162e6c7de347ff81a8a8b4fc0d35ec, State: Running, Role: LEADER
I20260925 22:45:58.783890 3794 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 39769 } }
I20260925 22:45:58.783946 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:58.770743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.0.201.190:39769)
0925 22:45:58.771140 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:58.771158 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:58.771200 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:45:58.771883 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:58.771901 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:58.772275 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:58.772547 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:58.773467 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:58.774457 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:58.775389 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:58.775910 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:58.776088 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:58.778804 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:58.778809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:58.778821 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:58.778846 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:58.780578 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:58.781155 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:58.781161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:58.781173 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:58.781278 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:58.781595 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:58.781598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:58.781600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:58.782053 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:58.782256 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:58.782563 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 22:45:58.782886 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":132,"threads_started":1}
I20260925 22:45:58.784547 3789 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:58.785514 3796 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7162e6c7de347ff81a8a8b4fc0d35ec. Latest consensus state: current_term: 1 leader_uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 39769 } } }
I20260925 22:45:58.785625 3796 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:58.785948 3795 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7162e6c7de347ff81a8a8b4fc0d35ec" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 39769 } } }
I20260925 22:45:58.786022 3795 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:58.786710 3803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:58.787091 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:45:58.789927 3803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:58.796404 3803 catalog_manager.cc:1347] Generated new cluster ID: d55ccd4141734441b48a46417d7c14ed
I20260925 22:45:58.796473 3803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:58.805073 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:58.799727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.0.201.190:39769)
0925 22:45:58.799850 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:58.799855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:58.800219 (+ 364us) server_negotiation.cc:407] Connection header received
0925 22:45:58.800435 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:58.800441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:58.800495 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:58.800557 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:58.801355 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:58.801901 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:58.802764 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:58.803025 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:58.803144 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:58.804128 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:58.804134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:58.804136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:58.804621 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:58.804649 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:58.804665 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:58.804775 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:58.804855 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:45:58.804913 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:45:58.814476 3803 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:58.815412 3803 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:58.820065 3803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7162e6c7de347ff81a8a8b4fc0d35ec: Generated new TSK 0
I20260925 22:45:58.820763 3803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:58.831938 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 3721
2026-09-25T22:45:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1409 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:58 dist-test-slave-vdtf krb5kdc[3822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:58 dist-test-slave-vdtf krb5kdc[3822](info): set up 2 sockets
Sep 25 22:45:58 dist-test-slave-vdtf krb5kdc[3822](info): commencing operation
krb5kdc: starting...
W20260925 22:45:59.875644 806 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:45:59 dist-test-slave-vdtf krb5kdc[3822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:46:00.011421 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42077
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:42077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:00.121336 3835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:00.121639 3835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:00.121714 3835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:00.125738 3835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:00.125830 3835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:00.125869 3835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:00.125927 3835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:00.125962 3835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:00.131137 3835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:42077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:00.132236 3835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:00.133311 3835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:00.141934 3841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:00.141997 3835 server_base.cc:1034] running on GCE node
W20260925 22:46:00.141953 3843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:00.141948 3840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:00.142900 3835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:00.144133 3835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:00.145352 3835 hybrid_clock.cc:648] HybridClock initialized: now 1790376360145298 us; error 85 us; skew 500 ppm
Sep 25 22:46:00 dist-test-slave-vdtf krb5kdc[3822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:00.148208 3835 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:46:00.149092 3835 webserver.cc:460] Webserver started at http://127.0.201.190:37533/ using document root <none> and password file <none>
I20260925 22:46:00.149667 3835 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:00.149760 3835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:00.149995 3835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:00.152153 3835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "ecf43727f5ab453c80f0d5f5a014293a"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "faa82d602a0555b9dfc9b62b35778d05"
server_key_iv: "aed11161dd225b9561c305d4a551e003"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.152741 3835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "ecf43727f5ab453c80f0d5f5a014293a"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "faa82d602a0555b9dfc9b62b35778d05"
server_key_iv: "aed11161dd225b9561c305d4a551e003"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.156433 3835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:46:00.158771 3850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:00.160156 3835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 22:46:00.160301 3835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "ecf43727f5ab453c80f0d5f5a014293a"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "faa82d602a0555b9dfc9b62b35778d05"
server_key_iv: "aed11161dd225b9561c305d4a551e003"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.160413 3835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:00.174266 3835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:00.177196 3835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:00.177412 3835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:00.184062 3835 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:42077
I20260925 22:46:00.184338 3902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:42077 every 8 connection(s)
I20260925 22:46:00.185106 3835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:00.187423 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 3835
I20260925 22:46:00.187546 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:00.187876 806 external_mini_cluster.cc:1428] Setting key d082074a002f7f93f5e39c011f5da72f
I20260925 22:46:00.188091 3903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:00.194099 3903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a: Bootstrap starting.
I20260925 22:46:00.196549 3903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a: Neither blocks nor log segments found. Creating new log.
Sep 25 22:46:00 dist-test-slave-vdtf krb5kdc[3822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:00.197422 3903 log.cc:824] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:00.199213 3903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a: No bootstrap required, opened a new log
I20260925 22:46:00.201977 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:00.189247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.0.201.190:42077)
0925 22:46:00.189823 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:00.189837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:00.189873 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:46:00.190690 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:00.190708 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:00.191017 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:00.191373 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:00.192171 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.193138 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.194004 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.194537 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.194709 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:00.197473 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:00.197477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:00.197490 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:00.197543 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:00.199338 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:00.199809 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:00.199814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:00.199826 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:00.199930 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:00.200237 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:00.200241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:00.200242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:00.200602 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:00.200811 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:00.201044 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 22:46:00.201313 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":126,"threads_started":1}
I20260925 22:46:00.202801 3903 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf43727f5ab453c80f0d5f5a014293a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42077 } }
I20260925 22:46:00.203033 3903 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:00.203115 3903 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf43727f5ab453c80f0d5f5a014293a, State: Initialized, Role: FOLLOWER
I20260925 22:46:00.203523 3903 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf43727f5ab453c80f0d5f5a014293a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42077 } }
I20260925 22:46:00.203670 3903 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:00.203769 3903 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:00.203889 3903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:00.204850 3903 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf43727f5ab453c80f0d5f5a014293a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42077 } }
I20260925 22:46:00.205175 3903 leader_election.cc:302] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecf43727f5ab453c80f0d5f5a014293a; no voters:
I20260925 22:46:00.205507 3903 leader_election.cc:288] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:00.205606 3908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:00.205832 3908 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [term 1 LEADER]: Becoming Leader. State: Replica: ecf43727f5ab453c80f0d5f5a014293a, State: Running, Role: LEADER
I20260925 22:46:00.206142 3908 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf43727f5ab453c80f0d5f5a014293a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42077 } }
I20260925 22:46:00.206699 3903 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:00.208029 3910 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf43727f5ab453c80f0d5f5a014293a. Latest consensus state: current_term: 1 leader_uuid: "ecf43727f5ab453c80f0d5f5a014293a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf43727f5ab453c80f0d5f5a014293a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42077 } } }
I20260925 22:46:00.208155 3910 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:00.208631 3909 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecf43727f5ab453c80f0d5f5a014293a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf43727f5ab453c80f0d5f5a014293a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42077 } } }
I20260925 22:46:00.208738 3909 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a [sys.catalog]: This master's current role is: LEADER
W20260925 22:46:00.211030 3924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:46:00.211125 3924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:46:00.211199 3917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:00.211948 3917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:00.217151 3917 catalog_manager.cc:1347] Generated new cluster ID: bcbff039dc6742139c7b4ef41627d4db
I20260925 22:46:00.217218 3917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:00.233839 3917 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:00.234798 3917 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:00.255853 3917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecf43727f5ab453c80f0d5f5a014293a: Generated new TSK 0
I20260925 22:46:00.256619 3917 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:46:00.290633 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42077
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:00.398504 3929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:00.398800 3929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:00.398877 3929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:00.402940 3929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:00.403038 3929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:00.403129 3929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:00.408528 3929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:00.409612 3929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:00.410805 3929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:00.420639 3937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:00.420589 3935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:00.420616 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:00.420804 3929 server_base.cc:1034] running on GCE node
I20260925 22:46:00.421269 3929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:00.421902 3929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:00.423077 3929 hybrid_clock.cc:648] HybridClock initialized: now 1790376360423040 us; error 49 us; skew 500 ppm
Sep 25 22:46:00 dist-test-slave-vdtf krb5kdc[3822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:00.430845 3929 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:46:00.431838 3929 webserver.cc:460] Webserver started at http://127.0.201.129:40141/ using document root <none> and password file <none>
I20260925 22:46:00.432421 3929 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:00.432483 3929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:00.432726 3929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:00.434849 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "6ea96ae69ca04b3d9e9e439a7da730b6"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "7a5cef6eb3ac311369ba2606f1a20618"
server_key_iv: "d5646a8338ebed39aeac358390b1898e"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.435428 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "6ea96ae69ca04b3d9e9e439a7da730b6"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "7a5cef6eb3ac311369ba2606f1a20618"
server_key_iv: "d5646a8338ebed39aeac358390b1898e"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.439173 3929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:46:00.441959 3944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:00.443274 3929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:00.443398 3929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "6ea96ae69ca04b3d9e9e439a7da730b6"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "7a5cef6eb3ac311369ba2606f1a20618"
server_key_iv: "d5646a8338ebed39aeac358390b1898e"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.443503 3929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:00.456646 3929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:00.459957 3929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:00.460176 3929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:00.460749 3929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:00.461537 3929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:00.461582 3929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:00.461654 3929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:00.461684 3929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:00.470449 3929 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:43245
I20260925 22:46:00.470484 4057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:43245 every 8 connection(s)
I20260925 22:46:00.471315 3929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:00.477020 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 3929
I20260925 22:46:00.477169 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:00.477447 806 external_mini_cluster.cc:1428] Setting key 5076c54499861b3943900c2cdb882c32
Sep 25 22:46:00 dist-test-slave-vdtf krb5kdc[3822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:00.485800 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:00.473075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:45817 (local address 127.0.201.190:42077)
0925 22:46:00.473236 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:00.473240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:00.474023 (+ 783us) server_negotiation.cc:407] Connection header received
0925 22:46:00.475414 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:00.475418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:00.475471 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:00.475551 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:00.476740 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.477379 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.478699 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.478935 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.479053 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:00.481786 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:00.481793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:00.481799 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:00.481831 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:00.483568 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:00.484129 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:00.484136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:00.484139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:00.484242 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:00.484579 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:00.484584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:00.484588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:00.484861 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:00.484980 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:00.485471 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 22:46:00.485608 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:46:00.486464 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:00.473449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42077 (local address 127.0.201.129:45817)
0925 22:46:00.473870 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:00.473911 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:00.475165 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:00.475774 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:00.475780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:00.476119 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:00.476581 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:00.476594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.477507 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.477510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:00.478567 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:00.478580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.479085 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.479089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:00.479265 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:00.479935 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:00.479961 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:00.481617 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:00.483752 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:00.483757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:00.483773 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:00.484017 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:00.484344 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:00.484346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:00.484348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:00.484468 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:00.484969 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:00.484974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:00.485188 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 22:46:00.485355 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 22:46:00.485593 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:46:00.487810 4058 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42077
I20260925 22:46:00.488088 4058 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:00.488564 4058 heartbeater.cc:511] Master 127.0.201.190:42077 requested a full tablet report, sending...
I20260925 22:46:00.490020 3867 ts_manager.cc:194] Registered new tserver with Master: 6ea96ae69ca04b3d9e9e439a7da730b6 (127.0.201.129:43245)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:46:00.491581 3867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:45817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:46:00.507133 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42077
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:00.616088 4063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:00.616395 4063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:00.616472 4063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:00.620502 4063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:00.620594 4063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:00.620682 4063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:00.625914 4063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:00.626993 4063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:00.628062 4063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:00.637658 4071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:00.637651 4069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:00.637756 4063 server_base.cc:1034] running on GCE node
W20260925 22:46:00.637672 4068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:00.638377 4063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:00.639050 4063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:00.640237 4063 hybrid_clock.cc:648] HybridClock initialized: now 1790376360640227 us; error 35 us; skew 500 ppm
Sep 25 22:46:00 dist-test-slave-vdtf krb5kdc[3822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:00.642892 4063 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:46:00.643797 4063 webserver.cc:460] Webserver started at http://127.0.201.130:40859/ using document root <none> and password file <none>
I20260925 22:46:00.644383 4063 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:00.644479 4063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:00.644711 4063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:00.646867 4063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "98bd5c04e6a04449beed01d33ec85158"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "1aab542a6f22dad94f49d237cedbfb56"
server_key_iv: "7167603b83a6144fabf3c7569428f47d"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.647483 4063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "98bd5c04e6a04449beed01d33ec85158"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "1aab542a6f22dad94f49d237cedbfb56"
server_key_iv: "7167603b83a6144fabf3c7569428f47d"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.651183 4063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:46:00.653631 4078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:00.654826 4063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:00.654966 4063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "98bd5c04e6a04449beed01d33ec85158"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "1aab542a6f22dad94f49d237cedbfb56"
server_key_iv: "7167603b83a6144fabf3c7569428f47d"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.655078 4063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:00.665628 4063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:00.668627 4063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:00.668824 4063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:00.669402 4063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:00.670205 4063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:00.670285 4063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:00.670356 4063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:00.670406 4063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:00.679464 4063 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:41891
I20260925 22:46:00.679481 4191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:41891 every 8 connection(s)
I20260925 22:46:00.680387 4063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:00.683087 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 4063
I20260925 22:46:00.683221 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:00.683530 806 external_mini_cluster.cc:1428] Setting key 30817e004508f0f36563f81de4f1d17c
Sep 25 22:46:00 dist-test-slave-vdtf krb5kdc[3822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:46:00.694818 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:00.682310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:55337 (local address 127.0.201.190:42077)
0925 22:46:00.682472 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:00.682477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:00.683451 (+ 974us) server_negotiation.cc:407] Connection header received
0925 22:46:00.685086 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:00.685089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:00.685140 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:00.685218 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:00.686349 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.686879 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.688090 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.688452 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.688554 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:00.691270 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:00.691274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:00.691277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:00.691308 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:00.692843 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:00.693440 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:00.693442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:00.693444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:00.693514 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:00.693821 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:00.693823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:00.693824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:00.693986 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:00.694059 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:00.694589 (+ 530us) server_negotiation.cc:299] Negotiation successful
0925 22:46:00.694686 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
I20260925 22:46:00.695940 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:00.682721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42077 (local address 127.0.201.130:55337)
0925 22:46:00.683301 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:00.683354 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:00.684873 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:00.685391 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:00.685397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:00.685743 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:00.686196 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:00.686208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.687008 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.687013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:00.687964 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:00.687972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.688588 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.688593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:00.688761 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:00.689416 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:00.689431 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:00.691097 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:00.693002 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:00.693009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:00.693026 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:00.693348 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:00.693598 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:00.693600 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:00.693601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:00.693740 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:00.694140 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:00.694147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:00.694381 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 22:46:00.694541 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 22:46:00.694888 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":148,"threads_started":1}
I20260925 22:46:00.697144 4192 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42077
I20260925 22:46:00.697388 4192 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:00.697872 4192 heartbeater.cc:511] Master 127.0.201.190:42077 requested a full tablet report, sending...
I20260925 22:46:00.698844 3867 ts_manager.cc:194] Registered new tserver with Master: 98bd5c04e6a04449beed01d33ec85158 (127.0.201.130:41891)
I20260925 22:46:00.699368 3867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:55337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:46:00.714747 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42077
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:00.822526 4197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:00.822795 4197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:00.822839 4197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:00.826887 4197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:00.826958 4197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:00.827016 4197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:00.832962 4197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:00.834153 4197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:00.835255 4197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:00.844947 4202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:00.844960 4205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:00.845046 4197 server_base.cc:1034] running on GCE node
W20260925 22:46:00.845027 4203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:00.845718 4197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:00.846436 4197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:00.847642 4197 hybrid_clock.cc:648] HybridClock initialized: now 1790376360847596 us; error 52 us; skew 500 ppm
Sep 25 22:46:00 dist-test-slave-vdtf krb5kdc[3822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:00.850427 4197 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:46:00.851280 4197 webserver.cc:460] Webserver started at http://127.0.201.131:39723/ using document root <none> and password file <none>
I20260925 22:46:00.851858 4197 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:00.851923 4197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:00.852090 4197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:00.854029 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "62cf086c2e4e4e40bf9717d0f89d61f9"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "d6813405f5dd01b18819a96c26f45763"
server_key_iv: "1a86a99b711805f60cbb4f3f6588a5c1"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.854538 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "62cf086c2e4e4e40bf9717d0f89d61f9"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "d6813405f5dd01b18819a96c26f45763"
server_key_iv: "1a86a99b711805f60cbb4f3f6588a5c1"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.858074 4197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:00.860641 4212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:00.861884 4197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:00.861976 4197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "62cf086c2e4e4e40bf9717d0f89d61f9"
format_stamp: "Formatted at 2026-09-25 22:46:00 on dist-test-slave-vdtf"
server_key: "d6813405f5dd01b18819a96c26f45763"
server_key_iv: "1a86a99b711805f60cbb4f3f6588a5c1"
server_key_version: "encryptionkey@0"
I20260925 22:46:00.862058 4197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:00.877417 4197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:00.880467 4197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:00.880620 4197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:00.881129 4197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:00.881878 4197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:00.881922 4197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:00.881960 4197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:00.881976 4197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:00.891753 4197 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:40151
I20260925 22:46:00.891800 4325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:40151 every 8 connection(s)
I20260925 22:46:00.892658 4197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:00.900667 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 4197
I20260925 22:46:00.900806 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:46:00.901096 806 external_mini_cluster.cc:1428] Setting key fcab1e2fdff72b9ba23383460cde7d49
Sep 25 22:46:00 dist-test-slave-vdtf krb5kdc[3822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:00.906435 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:00.894529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:44013 (local address 127.0.201.190:42077)
0925 22:46:00.894685 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:00.894689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:00.895502 (+ 813us) server_negotiation.cc:407] Connection header received
0925 22:46:00.897007 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:00.897011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:00.897069 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:00.897176 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:00.898332 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.898867 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.900106 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.900330 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.900469 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:00.903367 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:00.903370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:00.903372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:00.903394 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:00.904664 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:00.905098 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:00.905100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:00.905101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:00.905176 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:00.905466 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:00.905468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:00.905482 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:00.905636 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:00.905721 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:00.906159 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 22:46:00.906317 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:46:00.907347 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:00.894916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42077 (local address 127.0.201.131:44013)
0925 22:46:00.895348 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:00.895390 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:00.896774 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:00.897358 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:00.897365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:00.897712 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:00.898182 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:00.898196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.898996 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.899001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:00.899958 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:00.899968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.900485 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.900489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:00.900671 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:00.901340 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:00.901356 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:00.903207 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:00.904763 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:00.904767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:00.904777 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:00.905014 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:00.905266 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:00.905268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:00.905270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:00.905390 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:00.905733 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:00.905738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:00.905945 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 22:46:00.906112 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 22:46:00.906419 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":102,"threads_started":1}
I20260925 22:46:00.908490 4326 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42077
I20260925 22:46:00.908695 4326 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:00.909222 4326 heartbeater.cc:511] Master 127.0.201.190:42077 requested a full tablet report, sending...
I20260925 22:46:00.910176 3867 ts_manager.cc:194] Registered new tserver with Master: 62cf086c2e4e4e40bf9717d0f89d61f9 (127.0.201.131:40151)
I20260925 22:46:00.910714 3867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:44013
I20260925 22:46:00.914834 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:46:00 dist-test-slave-vdtf krb5kdc[3822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376360, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 22:46:00 dist-test-slave-vdtf krb5kdc[3822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376360, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:00.939067 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:00.931045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.0.201.190:42077)
0925 22:46:00.931205 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:00.931210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:00.931440 (+ 230us) server_negotiation.cc:407] Connection header received
0925 22:46:00.931518 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:00.931521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:00.931564 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:00.931637 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:00.932354 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.932879 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.933643 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:00.933845 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:00.933925 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:00.936307 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:00.936310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:00.936312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:00.936334 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:00.937553 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:00.937919 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:00.937922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:00.937923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:00.937991 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:00.938270 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:00.938272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:00.938273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:00.938471 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:00.938571 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:00.938777 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 22:46:00.938863 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 22:46:00.939496 3867 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36348
I20260925 22:46:00.940320 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 3929
I20260925 22:46:00.945647 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 4063
I20260925 22:46:00.951088 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 4197
I20260925 22:46:00.956251 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 3835
2026-09-25T22:46:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2125 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:46:01 dist-test-slave-vdtf krb5kdc[4338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:46:01 dist-test-slave-vdtf krb5kdc[4338](info): set up 2 sockets
Sep 25 22:46:01 dist-test-slave-vdtf krb5kdc[4338](info): commencing operation
krb5kdc: starting...
W20260925 22:46:02.007648 806 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:46:02 dist-test-slave-vdtf krb5kdc[4338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:46:02.127180 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:38373
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:02.234753 4351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:02.235069 4351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:02.235138 4351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:02.239117 4351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:02.239217 4351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:02.239265 4351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:02.239302 4351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:02.239338 4351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:02.244508 4351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:02.245576 4351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:02.246591 4351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:02.254843 4356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:02.254850 4357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:02.254899 4359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:02.255056 4351 server_base.cc:1034] running on GCE node
I20260925 22:46:02.255797 4351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:02.256803 4351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:02.257990 4351 hybrid_clock.cc:648] HybridClock initialized: now 1790376362257972 us; error 30 us; skew 500 ppm
Sep 25 22:46:02 dist-test-slave-vdtf krb5kdc[4338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:02.260730 4351 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:46:02.261660 4351 webserver.cc:460] Webserver started at http://127.0.201.190:34561/ using document root <none> and password file <none>
I20260925 22:46:02.262210 4351 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:02.262271 4351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:02.262499 4351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:02.264544 4351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "7279b9c7ccb841e197c19a76bb789a1c"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "a77b5ca23664543f6ae42e6ef489d91e"
server_key_iv: "d7a40dfb7bbb2b028b63ddb0ee539cde"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.265102 4351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "7279b9c7ccb841e197c19a76bb789a1c"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "a77b5ca23664543f6ae42e6ef489d91e"
server_key_iv: "d7a40dfb7bbb2b028b63ddb0ee539cde"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.268661 4351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:46:02.270875 4366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:02.272111 4351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:46:02.272241 4351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "7279b9c7ccb841e197c19a76bb789a1c"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "a77b5ca23664543f6ae42e6ef489d91e"
server_key_iv: "d7a40dfb7bbb2b028b63ddb0ee539cde"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.272343 4351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:02.283107 4351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:02.285806 4351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:02.286002 4351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:02.293227 4351 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:38373
I20260925 22:46:02.293256 4418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:38373 every 8 connection(s)
I20260925 22:46:02.294210 4351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:02.296926 4419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:02.302184 4419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c: Bootstrap starting.
I20260925 22:46:02.302862 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 4351
I20260925 22:46:02.302981 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:02.303263 806 external_mini_cluster.cc:1428] Setting key 8d5176881c4e7e1540ce0444dea3f334
I20260925 22:46:02.305056 4419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:02.305756 4419 log.cc:824] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:02.308009 4419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c: No bootstrap required, opened a new log
I20260925 22:46:02.312067 4419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7279b9c7ccb841e197c19a76bb789a1c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38373 } }
I20260925 22:46:02.312276 4419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:02.312337 4419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7279b9c7ccb841e197c19a76bb789a1c, State: Initialized, Role: FOLLOWER
Sep 25 22:46:02 dist-test-slave-vdtf krb5kdc[4338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:02.312824 4419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7279b9c7ccb841e197c19a76bb789a1c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38373 } }
I20260925 22:46:02.312954 4419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:02.313035 4419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:02.313126 4419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:02.314184 4419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7279b9c7ccb841e197c19a76bb789a1c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38373 } }
I20260925 22:46:02.314524 4419 leader_election.cc:302] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7279b9c7ccb841e197c19a76bb789a1c; no voters:
I20260925 22:46:02.314853 4419 leader_election.cc:288] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:02.314914 4424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:02.315105 4424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [term 1 LEADER]: Becoming Leader. State: Replica: 7279b9c7ccb841e197c19a76bb789a1c, State: Running, Role: LEADER
I20260925 22:46:02.315447 4424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7279b9c7ccb841e197c19a76bb789a1c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38373 } }
I20260925 22:46:02.316133 4419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:02.317214 4426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7279b9c7ccb841e197c19a76bb789a1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7279b9c7ccb841e197c19a76bb789a1c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38373 } } }
I20260925 22:46:02.317387 4426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:02.317222 4425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7279b9c7ccb841e197c19a76bb789a1c. Latest consensus state: current_term: 1 leader_uuid: "7279b9c7ccb841e197c19a76bb789a1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7279b9c7ccb841e197c19a76bb789a1c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38373 } } }
I20260925 22:46:02.317493 4425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:02.318926 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:02.304547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.0.201.190:38373)
0925 22:46:02.304947 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:02.304962 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:02.304996 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:46:02.305654 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:02.305677 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:02.306055 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:02.306437 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:02.307450 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.308888 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:02.309755 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.310481 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:02.310676 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:02.313336 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:02.313341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:02.313354 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:02.313380 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:02.315281 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:02.315834 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:02.315838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:02.315842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:02.315944 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:02.316242 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:02.316245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:02.316247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:02.316731 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:02.317534 (+ 803us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:02.317683 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 22:46:02.317972 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
W20260925 22:46:02.321470 4439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:46:02.321538 4439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:46:02.321614 4433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:02.322327 4433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:02.327590 4433 catalog_manager.cc:1347] Generated new cluster ID: c746b3f7c1c746fc8bad20c8744445a0
I20260925 22:46:02.327657 4433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:02.337767 4433 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:02.338682 4433 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:02.343894 4433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7279b9c7ccb841e197c19a76bb789a1c: Generated new TSK 0
I20260925 22:46:02.344728 4433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:46:02.378301 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:38373
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:02.486373 4445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:02.486613 4445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:02.486652 4445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:02.490594 4445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:02.490649 4445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:02.490705 4445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:02.495873 4445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:02.496876 4445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:02.497864 4445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:02.507074 4450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:02.507081 4451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:02.507062 4453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:02.507402 4445 server_base.cc:1034] running on GCE node
I20260925 22:46:02.507886 4445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:02.508538 4445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:02.509712 4445 hybrid_clock.cc:648] HybridClock initialized: now 1790376362509690 us; error 32 us; skew 500 ppm
Sep 25 22:46:02 dist-test-slave-vdtf krb5kdc[4338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:02.512598 4445 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:46:02.513458 4445 webserver.cc:460] Webserver started at http://127.0.201.129:40785/ using document root <none> and password file <none>
I20260925 22:46:02.514002 4445 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:02.514062 4445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:02.514233 4445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:02.516449 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "665ba17070fd40d5bda535211998f146"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "cc6f27db10f0e1b4415bd1c4a6ad59d0"
server_key_iv: "4d9aee8d60490d910817d0b74e821387"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.517011 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "665ba17070fd40d5bda535211998f146"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "cc6f27db10f0e1b4415bd1c4a6ad59d0"
server_key_iv: "4d9aee8d60490d910817d0b74e821387"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.521096 4445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:46:02.523361 4460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:02.524617 4445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:02.524708 4445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "665ba17070fd40d5bda535211998f146"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "cc6f27db10f0e1b4415bd1c4a6ad59d0"
server_key_iv: "4d9aee8d60490d910817d0b74e821387"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.524788 4445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:02.542172 4445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:02.545118 4445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:02.545280 4445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:02.545790 4445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:02.546543 4445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:02.546587 4445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:02.546625 4445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:02.546640 4445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:02.555287 4445 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:35653
I20260925 22:46:02.555315 4573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:35653 every 8 connection(s)
I20260925 22:46:02.556380 4445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:02.564328 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 4445
I20260925 22:46:02.564465 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:02.564765 806 external_mini_cluster.cc:1428] Setting key e6450df13adacb9e6b71fbee8c8773fa
Sep 25 22:46:02 dist-test-slave-vdtf krb5kdc[4338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:02.571105 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:02.558235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:53927 (local address 127.0.201.190:38373)
0925 22:46:02.558367 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:02.558371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:02.559230 (+ 859us) server_negotiation.cc:407] Connection header received
0925 22:46:02.560581 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:02.560584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:02.560630 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:02.560712 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:02.561833 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.562350 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:02.563525 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.563846 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:02.563929 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:02.566863 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:02.566866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:02.566867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:02.566890 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:02.568775 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:02.569273 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:02.569278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:02.569283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:02.569378 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:02.569705 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:02.569711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:02.569716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:02.569968 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:02.570079 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:02.570718 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 22:46:02.570855 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:46:02.571900 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:02.558645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:38373 (local address 127.0.201.129:53927)
0925 22:46:02.559076 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:02.559117 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:02.560388 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:02.560862 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:02.560869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:02.561220 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:02.561707 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:02.561720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.562496 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:02.562500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:02.563410 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:02.563420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.563984 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:02.563987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:02.564130 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:02.564896 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:02.564921 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:02.566679 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:02.568897 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:02.568902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:02.568915 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:02.569168 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:02.569478 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:02.569481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:02.569482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:02.569610 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:02.570104 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:02.570110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:02.570438 (+ 328us) client_negotiation.cc:769] Sending connection context
0925 22:46:02.570628 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 22:46:02.570899 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260925 22:46:02.573251 4574 heartbeater.cc:347] Connected to a master server at 127.0.201.190:38373
I20260925 22:46:02.573490 4574 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:02.573999 4574 heartbeater.cc:511] Master 127.0.201.190:38373 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:46:02.575790 4383 ts_manager.cc:194] Registered new tserver with Master: 665ba17070fd40d5bda535211998f146 (127.0.201.129:35653)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:46:02.577140 4383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:53927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:46:02.599433 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:38373
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:02.707718 4579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:02.708012 4579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:02.708091 4579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:02.712339 4579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:02.712423 4579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:02.712514 4579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:02.718281 4579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:02.719334 4579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:02.720444 4579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:02.729871 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:02.729911 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:02.729882 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:02.729882 4579 server_base.cc:1034] running on GCE node
I20260925 22:46:02.730484 4579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:02.731050 4579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:02.732201 4579 hybrid_clock.cc:648] HybridClock initialized: now 1790376362732189 us; error 27 us; skew 500 ppm
Sep 25 22:46:02 dist-test-slave-vdtf krb5kdc[4338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:02.734843 4579 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:46:02.735641 4579 webserver.cc:460] Webserver started at http://127.0.201.130:40015/ using document root <none> and password file <none>
I20260925 22:46:02.736223 4579 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:02.736284 4579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:02.736483 4579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:02.738417 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "e63f5699337844eca3dacc73b64789b6"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "ca959353cd5ca622ef2d766ab5afadcb"
server_key_iv: "fbb763eda81519f07f824a01adf5e56a"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.739022 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "e63f5699337844eca3dacc73b64789b6"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "ca959353cd5ca622ef2d766ab5afadcb"
server_key_iv: "fbb763eda81519f07f824a01adf5e56a"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.742583 4579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 22:46:02.744931 4594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:02.746066 4579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:46:02.746192 4579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "e63f5699337844eca3dacc73b64789b6"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "ca959353cd5ca622ef2d766ab5afadcb"
server_key_iv: "fbb763eda81519f07f824a01adf5e56a"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.746297 4579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:02.760486 4579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:02.763301 4579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:02.763490 4579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:02.764099 4579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:02.764889 4579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:02.764957 4579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:02.765020 4579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:02.765064 4579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:02.773809 4579 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:46245
I20260925 22:46:02.773833 4707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:46245 every 8 connection(s)
I20260925 22:46:02.774631 4579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:02.775468 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 4579
I20260925 22:46:02.775592 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:02.775942 806 external_mini_cluster.cc:1428] Setting key e0bfb979e7768c08c5075c409f8587e1
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
Sep 25 22:46:02 dist-test-slave-vdtf krb5kdc[4338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
I20260925 22:46:02.789530 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:02.776643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:51053 (local address 127.0.201.190:38373)
0925 22:46:02.776761 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:02.776765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:02.777729 (+ 964us) server_negotiation.cc:407] Connection header received
0925 22:46:02.779733 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:02.779738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:02.779801 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:02.779896 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:02.781004 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.781915 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:02.783052 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.783452 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:02.783567 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:02.786007 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:02.786010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:02.786012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:02.786034 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:02.787248 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:02.788011 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:02.788015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:02.788016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:02.788100 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:02.788409 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:02.788411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:02.788412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:02.788571 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:02.788646 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:02.789239 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 22:46:02.789333 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:46:02.790629 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:02.777070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:38373 (local address 127.0.201.130:51053)
0925 22:46:02.777540 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:02.777594 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:02.779505 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:02.780023 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:02.780029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:02.780381 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:02.780855 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:02.780868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.782047 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:02.782050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:02.782928 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:02.782937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.783594 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:02.783596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:02.783793 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:02.784424 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:02.784440 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:02.785853 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:02.787591 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:02.787599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:02.787616 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:02.787919 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:02.788199 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:02.788201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:02.788203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:02.788329 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:02.788756 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:02.788763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:02.789024 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 22:46:02.789204 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 22:46:02.789580 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260925 22:46:02.791877 4708 heartbeater.cc:347] Connected to a master server at 127.0.201.190:38373
I20260925 22:46:02.792148 4708 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:02.792619 4708 heartbeater.cc:511] Master 127.0.201.190:38373 requested a full tablet report, sending...
I20260925 22:46:02.793612 4383 ts_manager.cc:194] Registered new tserver with Master: e63f5699337844eca3dacc73b64789b6 (127.0.201.130:46245)
I20260925 22:46:02.794464 4383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:51053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:46:02.816655 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:38373
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:02.925077 4713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:02.925335 4713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:02.925377 4713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:02.929502 4713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:02.929561 4713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:02.929620 4713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:02.934749 4713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:02.935969 4713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:02.936995 4713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:02.946432 4721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:02.946556 4713 server_base.cc:1034] running on GCE node
W20260925 22:46:02.946421 4719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:02.946421 4718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:02.947132 4713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:02.947826 4713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:02.949020 4713 hybrid_clock.cc:648] HybridClock initialized: now 1790376362948996 us; error 36 us; skew 500 ppm
Sep 25 22:46:02 dist-test-slave-vdtf krb5kdc[4338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:02.951895 4713 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:46:02.952798 4713 webserver.cc:460] Webserver started at http://127.0.201.131:40663/ using document root <none> and password file <none>
I20260925 22:46:02.953378 4713 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:02.953469 4713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:02.953692 4713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:02.955664 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "3886b54a50d44cda8e39253467acebe2"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "cc38b99069aa497871760735d3582b73"
server_key_iv: "0303445dd08902a28d264c09ce9a8be5"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.956280 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "3886b54a50d44cda8e39253467acebe2"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "cc38b99069aa497871760735d3582b73"
server_key_iv: "0303445dd08902a28d264c09ce9a8be5"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.959857 4713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:46:02.962179 4728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:02.963375 4713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:02.963507 4713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "3886b54a50d44cda8e39253467acebe2"
format_stamp: "Formatted at 2026-09-25 22:46:02 on dist-test-slave-vdtf"
server_key: "cc38b99069aa497871760735d3582b73"
server_key_iv: "0303445dd08902a28d264c09ce9a8be5"
server_key_version: "encryptionkey@0"
I20260925 22:46:02.963614 4713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:02.979665 4713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:02.982494 4713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:02.982705 4713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:02.983291 4713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:02.984100 4713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:02.984185 4713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:02.984256 4713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:02.984290 4713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:02.992578 4713 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:39345
I20260925 22:46:02.992605 4841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:39345 every 8 connection(s)
I20260925 22:46:02.993451 4713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:03.003136 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 4713
I20260925 22:46:03.003258 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
Sep 25 22:46:02 dist-test-slave-vdtf krb5kdc[4338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:03.003595 806 external_mini_cluster.cc:1428] Setting key e61293ba438063525b5c2d1ff9720159
I20260925 22:46:03.007860 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:02.995359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:55045 (local address 127.0.201.190:38373)
0925 22:46:02.995514 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:02.995518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:02.996393 (+ 875us) server_negotiation.cc:407] Connection header received
0925 22:46:02.997866 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:02.997869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:02.997925 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:02.998025 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:02.999162 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:02.999844 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:03.001056 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:03.001301 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:03.001417 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:03.004065 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:03.004068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:03.004070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:03.004093 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:03.005839 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:03.006310 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:03.006315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:03.006316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:03.006405 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:03.006744 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:03.006749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:03.006751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:03.006951 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:03.007062 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:03.007546 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 22:46:03.007666 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:46:03.008605 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:02.995761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:38373 (local address 127.0.201.131:55045)
0925 22:46:02.996222 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:02.996264 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:02.997629 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:02.998200 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:02.998206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:02.998554 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:02.999016 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:02.999028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:03.000009 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:03.000013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:03.000927 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:03.000936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:03.001458 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:03.001462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:03.001637 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:03.002173 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:03.002189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:03.003888 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:03.005932 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:03.005938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:03.005951 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:03.006204 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:03.006503 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:03.006506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:03.006508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:03.006644 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:03.007069 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:03.007077 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:03.007311 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 22:46:03.007493 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 22:46:03.007835 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260925 22:46:03.009622 4842 heartbeater.cc:347] Connected to a master server at 127.0.201.190:38373
I20260925 22:46:03.009856 4842 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:03.010324 4842 heartbeater.cc:511] Master 127.0.201.190:38373 requested a full tablet report, sending...
I20260925 22:46:03.011278 4383 ts_manager.cc:194] Registered new tserver with Master: 3886b54a50d44cda8e39253467acebe2 (127.0.201.131:39345)
I20260925 22:46:03.011910 4383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:55045
I20260925 22:46:03.017514 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:46:03 dist-test-slave-vdtf krb5kdc[4338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:46:03 dist-test-slave-vdtf krb5kdc[4338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:46:03.042326 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:03.033409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.0.201.129:35653)
0925 22:46:03.033727 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:03.033743 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:03.033767 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:46:03.033814 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:03.033819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:03.034000 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:03.034119 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:03.034739 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:03.035531 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:03.036364 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:03.036733 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:03.036817 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:03.039072 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:03.039080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:03.039092 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:03.039115 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:03.040794 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:03.041160 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:03.041165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:03.041166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:03.041243 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:03.041458 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:03.041460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:03.041461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:03.041699 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:03.041860 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:03.041982 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 22:46:03.042128 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":110,"threads_started":1}
Sep 25 22:46:03 dist-test-slave-vdtf krb5kdc[4338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:03.052956 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:03.045176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.0.201.190:38373)
0925 22:46:03.045328 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:03.045333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:03.045378 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:46:03.045488 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:03.045490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:03.045529 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:03.045595 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:03.046110 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:03.046600 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:03.047292 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:03.047486 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:03.047557 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:03.049928 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:03.049931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:03.049932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:03.049955 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:03.051321 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:03.051735 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:03.051738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:03.051739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:03.051808 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:03.052060 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:03.052063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:03.052064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:03.052373 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:03.052482 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:03.052636 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 22:46:03.052768 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 22:46:03.053350 4383 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49232
I20260925 22:46:03.053921 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 4445
I20260925 22:46:03.060235 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 4579
I20260925 22:46:03.065017 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 4713
I20260925 22:46:03.070143 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 4351
2026-09-25T22:46:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2113 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:46:03 dist-test-slave-vdtf krb5kdc[4854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:46:03 dist-test-slave-vdtf krb5kdc[4854](info): set up 2 sockets
Sep 25 22:46:03 dist-test-slave-vdtf krb5kdc[4854](info): commencing operation
krb5kdc: starting...
W20260925 22:46:04.123754 806 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:46:04 dist-test-slave-vdtf krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:46:04.244179 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:46335
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:46335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:04.350616 4867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:04.350907 4867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:04.350986 4867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:04.355067 4867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:04.355154 4867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:04.355195 4867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:04.355252 4867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:04.355300 4867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:04.360419 4867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:46335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:46335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:04.361435 4867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:04.362427 4867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:04.370324 4875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:04.370451 4873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:04.370612 4867 server_base.cc:1034] running on GCE node
W20260925 22:46:04.372946 4872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:04.373450 4867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:04.374471 4867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:04.375654 4867 hybrid_clock.cc:648] HybridClock initialized: now 1790376364375640 us; error 32 us; skew 500 ppm
Sep 25 22:46:04 dist-test-slave-vdtf krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:04.378404 4867 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:46:04.379135 4867 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:04.379226 4867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:04.379448 4867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:04.381467 4867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf"
format_stamp: "Formatted at 2026-09-25 22:46:04 on dist-test-slave-vdtf"
server_key: "c59fe7efddd8fed828751aabc829bd40"
server_key_iv: "a4a17e667152e4a52c4654ed942f5d03"
server_key_version: "encryptionkey@0"
I20260925 22:46:04.382015 4867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf"
format_stamp: "Formatted at 2026-09-25 22:46:04 on dist-test-slave-vdtf"
server_key: "c59fe7efddd8fed828751aabc829bd40"
server_key_iv: "a4a17e667152e4a52c4654ed942f5d03"
server_key_version: "encryptionkey@0"
I20260925 22:46:04.385505 4867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:04.387797 4881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:04.388999 4867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:04.389142 4867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf"
format_stamp: "Formatted at 2026-09-25 22:46:04 on dist-test-slave-vdtf"
server_key: "c59fe7efddd8fed828751aabc829bd40"
server_key_iv: "a4a17e667152e4a52c4654ed942f5d03"
server_key_version: "encryptionkey@0"
I20260925 22:46:04.389248 4867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:04.405440 4867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:04.408061 4867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:04.408386 4867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:04.415210 4867 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:46335
I20260925 22:46:04.415503 4933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:46335 every 8 connection(s)
I20260925 22:46:04.415940 4867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:04.418507 4934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:04.419807 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 4867
I20260925 22:46:04.419946 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:04.420269 806 external_mini_cluster.cc:1428] Setting key efb5cdc5f7f2d4f2025f3081e203976a
I20260925 22:46:04.424553 4934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf: Bootstrap starting.
I20260925 22:46:04.427003 4934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:04.427912 4934 log.cc:824] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:46:04 dist-test-slave-vdtf krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:04.429893 4934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf: No bootstrap required, opened a new log
I20260925 22:46:04.434463 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:04.421634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.0.201.190:46335)
0925 22:46:04.422013 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:04.422031 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:04.422073 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:46:04.422676 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:04.422692 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:04.422993 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:04.423251 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:04.424188 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:04.425170 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:04.426326 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:04.426762 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:04.426873 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:04.429521 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:04.429527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:04.429542 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:04.429573 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:04.431486 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:04.431944 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:04.431949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:04.431955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:04.432097 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:04.432364 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:04.432369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:04.432372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:04.432883 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:04.433067 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:04.433255 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 22:46:04.433548 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
I20260925 22:46:04.434512 4934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46335 } }
I20260925 22:46:04.434754 4934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:04.434818 4934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b3c22af7cb4d29b0bd5aaec372ebdf, State: Initialized, Role: FOLLOWER
I20260925 22:46:04.435232 4934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46335 } }
I20260925 22:46:04.435379 4934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:04.435477 4934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:04.435575 4934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:04.436492 4934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46335 } }
I20260925 22:46:04.436810 4934 leader_election.cc:302] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73b3c22af7cb4d29b0bd5aaec372ebdf; no voters:
I20260925 22:46:04.437099 4934 leader_election.cc:288] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:04.437238 4939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:04.437476 4939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [term 1 LEADER]: Becoming Leader. State: Replica: 73b3c22af7cb4d29b0bd5aaec372ebdf, State: Running, Role: LEADER
I20260925 22:46:04.437810 4939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46335 } }
I20260925 22:46:04.438220 4934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:04.439335 4940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46335 } } }
I20260925 22:46:04.439463 4940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:04.439886 4941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73b3c22af7cb4d29b0bd5aaec372ebdf. Latest consensus state: current_term: 1 leader_uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b3c22af7cb4d29b0bd5aaec372ebdf" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46335 } } }
I20260925 22:46:04.439903 4948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:04.440091 4941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:04.442747 4948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:04.447831 4948 catalog_manager.cc:1347] Generated new cluster ID: b9e945bd28324de58be368540b4c8bf1
I20260925 22:46:04.447903 4948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:04.463281 4948 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:04.464219 4948 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:04.469168 4948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf: Generated new TSK 0
I20260925 22:46:04.469833 4948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:46:04.498817 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46335
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:04.607892 4960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:04.608135 4960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:04.608218 4960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:04.612221 4960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:04.612275 4960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:04.612360 4960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:04.617470 4960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:04.618497 4960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:04.619488 4960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:04.628886 4965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:04.628893 4968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:04.628907 4966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:04.629231 4960 server_base.cc:1034] running on GCE node
I20260925 22:46:04.629696 4960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:04.630367 4960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:04.631543 4960 hybrid_clock.cc:648] HybridClock initialized: now 1790376364631510 us; error 43 us; skew 500 ppm
Sep 25 22:46:04 dist-test-slave-vdtf krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:04.634433 4960 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:46:04.635255 4960 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:04.635352 4960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:04.635591 4960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:04.637894 4960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "7f74a2de8da844919f195efe63519643"
format_stamp: "Formatted at 2026-09-25 22:46:04 on dist-test-slave-vdtf"
server_key: "23eaa9001a41a0f52c252f369266772b"
server_key_iv: "f13786ecc159e359788db3a925d7edb9"
server_key_version: "encryptionkey@0"
I20260925 22:46:04.638492 4960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "7f74a2de8da844919f195efe63519643"
format_stamp: "Formatted at 2026-09-25 22:46:04 on dist-test-slave-vdtf"
server_key: "23eaa9001a41a0f52c252f369266772b"
server_key_iv: "f13786ecc159e359788db3a925d7edb9"
server_key_version: "encryptionkey@0"
I20260925 22:46:04.642288 4960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:46:04.644687 4974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:04.645933 4960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:04.646080 4960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "7f74a2de8da844919f195efe63519643"
format_stamp: "Formatted at 2026-09-25 22:46:04 on dist-test-slave-vdtf"
server_key: "23eaa9001a41a0f52c252f369266772b"
server_key_iv: "f13786ecc159e359788db3a925d7edb9"
server_key_version: "encryptionkey@0"
I20260925 22:46:04.646188 4960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:04.664733 4960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:04.667326 4960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:04.667520 4960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:04.668095 4960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:04.668870 4960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:04.668941 4960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:04.669004 4960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:04.669056 4960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:04.677800 4960 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:33619
I20260925 22:46:04.677817 5087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:33619 every 8 connection(s)
I20260925 22:46:04.678363 4960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:04.685585 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 4960
I20260925 22:46:04.685720 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:04.686014 806 external_mini_cluster.cc:1428] Setting key 09c0832a306b8adf060f051cb84c5d01
Sep 25 22:46:04 dist-test-slave-vdtf krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:04.691890 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:04.679970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:55485 (local address 127.0.201.190:46335)
0925 22:46:04.680091 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:04.680095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:04.681023 (+ 928us) server_negotiation.cc:407] Connection header received
0925 22:46:04.682292 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:04.682300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:04.682344 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:04.682420 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:04.683441 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:04.684036 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:04.685109 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:04.685360 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:04.685463 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:04.688120 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:04.688127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:04.688133 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:04.688166 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:04.689661 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:04.690265 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:04.690267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:04.690269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:04.690348 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:04.690735 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:04.690738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:04.690739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:04.690899 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:04.690972 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:04.691635 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 22:46:04.691777 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:46:04.693040 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:04.680390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:46335 (local address 127.0.201.129:55485)
0925 22:46:04.680864 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:04.680914 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:04.682109 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:04.682542 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:04.682549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:04.682872 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:04.683312 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:04.683324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:04.684138 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:04.684141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:04.685010 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:04.685018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:04.685489 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:04.685492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:04.685626 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:04.686272 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:04.686298 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:04.687965 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:04.689780 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:04.689788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:04.689804 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:04.690165 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:04.690457 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:04.690463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:04.690468 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:04.690645 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:04.691002 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:04.691011 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:04.691321 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 22:46:04.691552 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 22:46:04.691920 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":131,"threads_started":1}
I20260925 22:46:04.694304 5088 heartbeater.cc:347] Connected to a master server at 127.0.201.190:46335
I20260925 22:46:04.694554 5088 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:04.695066 5088 heartbeater.cc:511] Master 127.0.201.190:46335 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:46:04.696739 4898 ts_manager.cc:194] Registered new tserver with Master: 7f74a2de8da844919f195efe63519643 (127.0.201.129:33619)
I20260925 22:46:04.697831 4898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:55485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:46:04.715574 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46335
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:04.825183 5093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:04.825431 5093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:04.825515 5093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:04.829490 5093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:04.829546 5093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:04.829603 5093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:04.834833 5093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:04.835975 5093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:04.836989 5093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:04.846611 5098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:04.846616 5099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:04.846757 5093 server_base.cc:1034] running on GCE node
W20260925 22:46:04.846625 5101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:04.847451 5093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:04.848163 5093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:04.849331 5093 hybrid_clock.cc:648] HybridClock initialized: now 1790376364849299 us; error 45 us; skew 500 ppm
Sep 25 22:46:04 dist-test-slave-vdtf krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:04.852403 5093 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:46:04.853168 5093 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:04.853230 5093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:04.853520 5093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:04.855643 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "00e243e1b4404ffeaca6ed7733f2cd93"
format_stamp: "Formatted at 2026-09-25 22:46:04 on dist-test-slave-vdtf"
server_key: "21d3b8c29aba0b81126baf7506c43627"
server_key_iv: "c87fb98cc32623416cefa8efb87fca63"
server_key_version: "encryptionkey@0"
I20260925 22:46:04.856281 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "00e243e1b4404ffeaca6ed7733f2cd93"
format_stamp: "Formatted at 2026-09-25 22:46:04 on dist-test-slave-vdtf"
server_key: "21d3b8c29aba0b81126baf7506c43627"
server_key_iv: "c87fb98cc32623416cefa8efb87fca63"
server_key_version: "encryptionkey@0"
I20260925 22:46:04.859993 5093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:46:04.862411 5107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:04.863662 5093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:04.863867 5093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "00e243e1b4404ffeaca6ed7733f2cd93"
format_stamp: "Formatted at 2026-09-25 22:46:04 on dist-test-slave-vdtf"
server_key: "21d3b8c29aba0b81126baf7506c43627"
server_key_iv: "c87fb98cc32623416cefa8efb87fca63"
server_key_version: "encryptionkey@0"
I20260925 22:46:04.863976 5093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:04.881145 5093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:04.884490 5093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:04.884697 5093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:04.885289 5093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:04.886112 5093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:04.886163 5093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:04.886202 5093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:04.886271 5093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:04.895434 5093 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:41143
I20260925 22:46:04.895452 5220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:41143 every 8 connection(s)
I20260925 22:46:04.896087 5093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:04.902355 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 5093
I20260925 22:46:04.902478 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:04.902833 806 external_mini_cluster.cc:1428] Setting key 0bf992e8b09021ab3841855f2cee1c0d
Sep 25 22:46:04 dist-test-slave-vdtf krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:04.911290 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:04.897789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:35037 (local address 127.0.201.190:46335)
0925 22:46:04.897924 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:04.897927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:04.898681 (+ 754us) server_negotiation.cc:407] Connection header received
0925 22:46:04.900072 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:04.900076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:04.900130 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:04.900221 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:04.901813 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:04.902589 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:04.904161 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:04.904474 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:04.904582 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:04.907457 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:04.907461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:04.907463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:04.907496 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:04.909055 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:04.909643 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:04.909645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:04.909647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:04.909718 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:04.910128 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:04.910131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:04.910132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:04.910327 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:04.910443 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:04.911071 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 22:46:04.911165 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:46:04.912508 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:04.898130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:46335 (local address 127.0.201.130:35037)
0925 22:46:04.898542 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:04.898582 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:04.899869 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:04.900416 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:04.900427 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:04.900913 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:04.901641 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:04.901662 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:04.902737 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:04.902742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:04.904048 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:04.904058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:04.904631 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:04.904635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:04.904822 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:04.905420 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:04.905445 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:04.907291 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:04.909166 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:04.909172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:04.909188 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:04.909545 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:04.909820 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:04.909826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:04.909831 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:04.910025 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:04.910435 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:04.910444 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:04.910793 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 22:46:04.911032 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 22:46:04.911387 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:46:04.913812 5221 heartbeater.cc:347] Connected to a master server at 127.0.201.190:46335
I20260925 22:46:04.914040 5221 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:04.914489 5221 heartbeater.cc:511] Master 127.0.201.190:46335 requested a full tablet report, sending...
I20260925 22:46:04.915565 4898 ts_manager.cc:194] Registered new tserver with Master: 00e243e1b4404ffeaca6ed7733f2cd93 (127.0.201.130:41143)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
I20260925 22:46:04.916409 4898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:35037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:46:04.931918 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46335
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:05.043028 5226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:05.043262 5226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:05.043305 5226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:05.047245 5226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:05.047307 5226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:05.047367 5226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:05.052708 5226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:05.053649 5226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:05.054571 5226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:05.063609 5232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:05.063642 5234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:05.063819 5226 server_base.cc:1034] running on GCE node
W20260925 22:46:05.063611 5231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:05.064278 5226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:05.064896 5226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:05.066072 5226 hybrid_clock.cc:648] HybridClock initialized: now 1790376365066058 us; error 27 us; skew 500 ppm
Sep 25 22:46:05 dist-test-slave-vdtf krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:05.068899 5226 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:46:05.069617 5226 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:05.069718 5226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:05.069939 5226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:05.071907 5226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1"
format_stamp: "Formatted at 2026-09-25 22:46:05 on dist-test-slave-vdtf"
server_key: "1dd24da0b57e011f3ae634a9c95c532b"
server_key_iv: "efd6620e5eb2ffb052b9e361c978e36d"
server_key_version: "encryptionkey@0"
I20260925 22:46:05.072455 5226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1"
format_stamp: "Formatted at 2026-09-25 22:46:05 on dist-test-slave-vdtf"
server_key: "1dd24da0b57e011f3ae634a9c95c532b"
server_key_iv: "efd6620e5eb2ffb052b9e361c978e36d"
server_key_version: "encryptionkey@0"
I20260925 22:46:05.075829 5226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:46:05.078064 5240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:05.079274 5226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:05.079397 5226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1"
format_stamp: "Formatted at 2026-09-25 22:46:05 on dist-test-slave-vdtf"
server_key: "1dd24da0b57e011f3ae634a9c95c532b"
server_key_iv: "efd6620e5eb2ffb052b9e361c978e36d"
server_key_version: "encryptionkey@0"
I20260925 22:46:05.079499 5226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:05.092850 5226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:05.095836 5226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:05.096024 5226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:05.096576 5226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:05.097434 5226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:05.097502 5226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:05.097565 5226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:05.097615 5226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:05.106146 5226 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:38491
I20260925 22:46:05.106163 5353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:38491 every 8 connection(s)
I20260925 22:46:05.106714 5226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:05.107766 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 5226
I20260925 22:46:05.107852 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:46:05.108212 806 external_mini_cluster.cc:1428] Setting key 37f8678a9f542b3510cc1e83e3767901
Sep 25 22:46:05 dist-test-slave-vdtf krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:05.119936 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.108570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:40207 (local address 127.0.201.190:46335)
0925 22:46:05.108719 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:05.108723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:05.109716 (+ 993us) server_negotiation.cc:407] Connection header received
0925 22:46:05.111058 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:05.111060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:05.111112 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:05.111179 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:05.112313 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.112830 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.113963 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.114174 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.114262 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.116640 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:05.116643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:05.116645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:05.116668 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:05.118128 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:05.118562 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:05.118565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:05.118566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:05.118635 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:05.118939 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:05.118942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:05.118943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:05.119132 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:05.119228 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:05.119655 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 22:46:05.119818 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:46:05.120793 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.109085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:46335 (local address 127.0.201.131:40207)
0925 22:46:05.109558 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:05.109608 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:05.110867 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:05.111309 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:05.111315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:05.111645 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:05.112174 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.112187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.112961 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.112964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.113838 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.113848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.114333 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.114336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.114477 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.115034 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:05.115050 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:05.116493 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:05.118218 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:05.118222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:05.118233 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:05.118481 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:05.118710 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:05.118712 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:05.118713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:05.118841 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:05.119224 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:05.119228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:05.119452 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 22:46:05.119598 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 22:46:05.119900 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260925 22:46:05.122016 5354 heartbeater.cc:347] Connected to a master server at 127.0.201.190:46335
I20260925 22:46:05.122274 5354 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:05.122700 5354 heartbeater.cc:511] Master 127.0.201.190:46335 requested a full tablet report, sending...
I20260925 22:46:05.123626 4898 ts_manager.cc:194] Registered new tserver with Master: 97df6e24ebb84e039a7a8fbe3ebc2aa1 (127.0.201.131:38491)
I20260925 22:46:05.124197 4898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:40207
I20260925 22:46:05.133565 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:46:05.142369 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.135429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.0.201.190:46335)
0925 22:46:05.135547 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:05.135551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:05.135799 (+ 248us) server_negotiation.cc:407] Connection header received
0925 22:46:05.135910 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:05.135912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:05.135949 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:05.136013 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:05.136645 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.137141 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.137897 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.138093 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.138167 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.139538 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:05.139541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:05.139543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:05.139564 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:05.141028 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:05.141398 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:05.141401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:05.141402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:05.141469 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:05.141683 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:05.141685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:05.141686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:05.141896 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:05.141995 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:05.142174 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 22:46:05.142257 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:46:05.145012 4898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:46:05.147993 4898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:05.162206 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.156280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:38491 (local address 127.0.0.1:39310)
0925 22:46:05.156589 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:05.156635 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:05.156750 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:05.157672 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:05.157677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:05.157697 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:05.157868 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.157877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.159391 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.159393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.160763 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.160772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.161647 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.161651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.161760 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.161794 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:46:05.161937 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 22:46:05.162060 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260925 22:46:05.162214 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.156759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.0.201.131:38491)
0925 22:46:05.157139 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:05.157154 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:05.157179 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:46:05.157254 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:05.157259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:05.157498 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:05.157686 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:05.158100 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.159272 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.160904 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.161528 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.161651 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.161796 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:05.161967 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 22:46:05.162060 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":158,"threads_started":1}
I20260925 22:46:05.162488 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.157073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:41143 (local address 127.0.0.1:54260)
0925 22:46:05.157529 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:05.157545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:05.157631 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:05.158364 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:05.158366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:05.158381 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:05.158456 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.158460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.159698 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.159701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.161038 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.161045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.162125 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.162128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.162223 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.162245 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:46:05.162293 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 22:46:05.162355 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":86,"threads_started":1}
I20260925 22:46:05.163060 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.157242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54260 (local address 127.0.201.130:41143)
0925 22:46:05.157886 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:05.157897 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:05.157947 (+ 50us) server_negotiation.cc:407] Connection header received
0925 22:46:05.158023 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:05.158028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:05.158229 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:05.158389 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:05.158532 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.159564 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.161157 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.161941 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.162653 (+ 712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.162786 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:05.162832 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 22:46:05.162917 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":99,"threads_started":1}
I20260925 22:46:05.165184 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.156361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:33619 (local address 127.0.0.1:38262)
0925 22:46:05.157450 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:05.157465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:05.157546 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:05.160393 (+ 2847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:05.160396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:05.160420 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:05.160528 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.160535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.162104 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.162107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.163805 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.163817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.164808 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.164812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.164905 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.164927 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:46:05.165004 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:46:05.165065 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"thread_start_us":85,"threads_started":1}
I20260925 22:46:05.165271 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.157174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.0.201.129:33619)
0925 22:46:05.159847 (+ 2673us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:05.159853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:05.159871 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:46:05.159927 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:05.159930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:05.160057 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:05.160236 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:05.160659 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.161930 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.163931 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.164670 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.164788 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.164903 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:05.165043 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 22:46:05.165128 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2590,"thread_start_us":98,"threads_started":1}
I20260925 22:46:05.165722 5288 tablet_service.cc:1467] Processing CreateTablet for tablet a8c8caad60f448c69a004f36af522297 (DEFAULT_TABLE table=test-table [id=5002fc6f0ce841d8910824378201f75b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:05.165727 5155 tablet_service.cc:1467] Processing CreateTablet for tablet a8c8caad60f448c69a004f36af522297 (DEFAULT_TABLE table=test-table [id=5002fc6f0ce841d8910824378201f75b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:05.166594 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c8caad60f448c69a004f36af522297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:05.166601 5288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c8caad60f448c69a004f36af522297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:05.169982 5022 tablet_service.cc:1467] Processing CreateTablet for tablet a8c8caad60f448c69a004f36af522297 (DEFAULT_TABLE table=test-table [id=5002fc6f0ce841d8910824378201f75b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:05.171010 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c8caad60f448c69a004f36af522297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:05.172982 5371 tablet_bootstrap.cc:492] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: Bootstrap starting.
I20260925 22:46:05.174363 5372 tablet_bootstrap.cc:492] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: Bootstrap starting.
I20260925 22:46:05.175251 5371 tablet_bootstrap.cc:654] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:05.176446 5372 tablet_bootstrap.cc:654] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:05.176590 5371 log.cc:824] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:05.177173 5373 tablet_bootstrap.cc:492] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643: Bootstrap starting.
I20260925 22:46:05.177661 5372 log.cc:824] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:05.178556 5371 tablet_bootstrap.cc:492] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: No bootstrap required, opened a new log
I20260925 22:46:05.178745 5371 ts_tablet_manager.cc:1397] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:46:05.179106 5373 tablet_bootstrap.cc:654] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:05.179610 5372 tablet_bootstrap.cc:492] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: No bootstrap required, opened a new log
I20260925 22:46:05.179802 5372 ts_tablet_manager.cc:1397] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:46:05.180214 5373 log.cc:824] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:05.181612 5373 tablet_bootstrap.cc:492] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643: No bootstrap required, opened a new log
I20260925 22:46:05.181955 5373 ts_tablet_manager.cc:1397] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:46:05.182508 5371 raft_consensus.cc:348] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 } } peers { permanent_uuid: "7f74a2de8da844919f195efe63519643" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33619 } } peers { permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 } }
I20260925 22:46:05.182776 5371 raft_consensus.cc:374] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:05.182829 5371 raft_consensus.cc:729] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df6e24ebb84e039a7a8fbe3ebc2aa1, State: Initialized, Role: FOLLOWER
I20260925 22:46:05.183264 5371 consensus_queue.cc:260] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 } } peers { permanent_uuid: "7f74a2de8da844919f195efe63519643" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33619 } } peers { permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 } }
I20260925 22:46:05.183547 5372 raft_consensus.cc:348] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 } } peers { permanent_uuid: "7f74a2de8da844919f195efe63519643" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33619 } } peers { permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 } }
I20260925 22:46:05.183844 5372 raft_consensus.cc:374] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:05.183899 5372 raft_consensus.cc:729] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e243e1b4404ffeaca6ed7733f2cd93, State: Initialized, Role: FOLLOWER
I20260925 22:46:05.184108 5354 heartbeater.cc:503] Master 127.0.201.190:46335 was elected leader, sending a full tablet report...
I20260925 22:46:05.184290 5372 consensus_queue.cc:260] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 } } peers { permanent_uuid: "7f74a2de8da844919f195efe63519643" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33619 } } peers { permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 } }
I20260925 22:46:05.185035 5221 heartbeater.cc:503] Master 127.0.201.190:46335 was elected leader, sending a full tablet report...
I20260925 22:46:05.185160 5372 ts_tablet_manager.cc:1428] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:46:05.185895 5371 ts_tablet_manager.cc:1428] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:46:05.186961 5373 raft_consensus.cc:348] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 } } peers { permanent_uuid: "7f74a2de8da844919f195efe63519643" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33619 } } peers { permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 } }
I20260925 22:46:05.187177 5373 raft_consensus.cc:374] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:05.187240 5373 raft_consensus.cc:729] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f74a2de8da844919f195efe63519643, State: Initialized, Role: FOLLOWER
I20260925 22:46:05.187615 5373 consensus_queue.cc:260] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 } } peers { permanent_uuid: "7f74a2de8da844919f195efe63519643" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33619 } } peers { permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 } }
I20260925 22:46:05.188294 5088 heartbeater.cc:503] Master 127.0.201.190:46335 was elected leader, sending a full tablet report...
I20260925 22:46:05.188571 5373 ts_tablet_manager.cc:1428] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
W20260925 22:46:05.357551 5355 tablet.cc:2369] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:05.374081 5379 raft_consensus.cc:482] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:05.374439 5379 raft_consensus.cc:504] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 } } peers { permanent_uuid: "7f74a2de8da844919f195efe63519643" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33619 } } peers { permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 } }
I20260925 22:46:05.375609 5379 leader_election.cc:288] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df6e24ebb84e039a7a8fbe3ebc2aa1 (127.0.201.131:38491), 00e243e1b4404ffeaca6ed7733f2cd93 (127.0.201.130:41143)
I20260925 22:46:05.380786 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.375969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:46431 (local address 127.0.201.131:38491)
0925 22:46:05.376128 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:05.376134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:05.376301 (+ 167us) server_negotiation.cc:407] Connection header received
0925 22:46:05.376493 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:05.376497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:05.376548 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:05.376633 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:05.376984 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.377772 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.379428 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.380063 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.380196 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.380247 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:05.380589 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 22:46:05.380657 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:46:05.380786 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.376442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:55563 (local address 127.0.201.130:41143)
0925 22:46:05.376557 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:05.376562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:05.376722 (+ 160us) server_negotiation.cc:407] Connection header received
0925 22:46:05.376810 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:05.376812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:05.376849 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:05.376907 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:05.377151 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.377866 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.379428 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.380063 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.380197 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.380247 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:05.380589 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 22:46:05.380657 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:46:05.380957 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.376314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:41143 (local address 127.0.201.129:55563)
0925 22:46:05.376623 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:05.376637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:05.376720 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:05.376940 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:05.376945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:05.376963 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:05.377055 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.377060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.377964 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.377967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.379303 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.379314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.380222 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.380225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.380431 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.380467 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:46:05.380566 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:46:05.380741 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":158,"thread_start_us":130,"threads_started":1}
I20260925 22:46:05.381333 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.375902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:38491 (local address 127.0.201.129:46431)
0925 22:46:05.376188 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:05.376203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:05.376359 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:05.376673 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:05.376678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:05.376699 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:05.376859 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.376864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.377887 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.377889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.379303 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:05.379314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.380215 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.380220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:05.380390 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.380467 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 22:46:05.380566 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:46:05.380734 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":141,"spinlock_wait_cycles":1920,"thread_start_us":97,"threads_started":1}
I20260925 22:46:05.381717 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c8caad60f448c69a004f36af522297" candidate_uuid: "7f74a2de8da844919f195efe63519643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" is_pre_election: true
I20260925 22:46:05.382053 5308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c8caad60f448c69a004f36af522297" candidate_uuid: "7f74a2de8da844919f195efe63519643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" is_pre_election: true
I20260925 22:46:05.382117 5175 raft_consensus.cc:2454] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f74a2de8da844919f195efe63519643 in term 0.
I20260925 22:46:05.382366 5308 raft_consensus.cc:2454] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f74a2de8da844919f195efe63519643 in term 0.
I20260925 22:46:05.382642 4975 leader_election.cc:302] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e243e1b4404ffeaca6ed7733f2cd93, 7f74a2de8da844919f195efe63519643; no voters:
I20260925 22:46:05.382922 5379 raft_consensus.cc:2781] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:05.383003 5379 raft_consensus.cc:482] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:05.383081 5379 raft_consensus.cc:3037] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:05.384289 5379 raft_consensus.cc:504] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 } } peers { permanent_uuid: "7f74a2de8da844919f195efe63519643" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33619 } } peers { permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 } }
I20260925 22:46:05.384653 5379 leader_election.cc:288] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [CANDIDATE]: Term 1 election: Requested vote from peers 97df6e24ebb84e039a7a8fbe3ebc2aa1 (127.0.201.131:38491), 00e243e1b4404ffeaca6ed7733f2cd93 (127.0.201.130:41143)
I20260925 22:46:05.385018 5308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c8caad60f448c69a004f36af522297" candidate_uuid: "7f74a2de8da844919f195efe63519643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1"
I20260925 22:46:05.385058 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c8caad60f448c69a004f36af522297" candidate_uuid: "7f74a2de8da844919f195efe63519643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e243e1b4404ffeaca6ed7733f2cd93"
I20260925 22:46:05.385130 5308 raft_consensus.cc:3037] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:05.385195 5175 raft_consensus.cc:3037] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:05.386240 5175 raft_consensus.cc:2454] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f74a2de8da844919f195efe63519643 in term 1.
I20260925 22:46:05.386240 5308 raft_consensus.cc:2454] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f74a2de8da844919f195efe63519643 in term 1.
I20260925 22:46:05.386562 4975 leader_election.cc:302] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f74a2de8da844919f195efe63519643, 97df6e24ebb84e039a7a8fbe3ebc2aa1; no voters:
I20260925 22:46:05.386730 5379 raft_consensus.cc:2781] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:05.386957 5379 raft_consensus.cc:686] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [term 1 LEADER]: Becoming Leader. State: Replica: 7f74a2de8da844919f195efe63519643, State: Running, Role: LEADER
I20260925 22:46:05.387290 5379 consensus_queue.cc:237] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 } } peers { permanent_uuid: "7f74a2de8da844919f195efe63519643" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33619 } } peers { permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 } }
I20260925 22:46:05.390424 4898 catalog_manager.cc:5495] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f74a2de8da844919f195efe63519643 (127.0.201.129). New cstate: current_term: 1 leader_uuid: "7f74a2de8da844919f195efe63519643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f74a2de8da844919f195efe63519643" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 } health_report { overall_health: UNKNOWN } } }
W20260925 22:46:05.397037 5222 tablet.cc:2369] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:46:05.429518 5089 tablet.cc:2369] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:05.457000 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.453322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.0.201.129:33619)
0925 22:46:05.453465 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:05.453469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:05.453485 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:46:05.453608 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:05.453610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:05.453667 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:05.453782 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:05.454046 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.454553 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.455416 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:05.455636 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:05.455791 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:05.456153 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:05.456628 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:05.456763 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:05.456798 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 22:46:05.456854 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:46:05.462568 5308 raft_consensus.cc:1261] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1 [term 1 FOLLOWER]: Refusing update from remote peer 7f74a2de8da844919f195efe63519643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:05.462567 5175 raft_consensus.cc:1261] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93 [term 1 FOLLOWER]: Refusing update from remote peer 7f74a2de8da844919f195efe63519643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:05.463253 5382 consensus_queue.cc:1035] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e243e1b4404ffeaca6ed7733f2cd93" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:05.463507 5379 consensus_queue.cc:1035] T a8c8caad60f448c69a004f36af522297 P 7f74a2de8da844919f195efe63519643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df6e24ebb84e039a7a8fbe3ebc2aa1" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:05.470208 5387 mvcc.cc:205] Tried to move back new op lower bound from 7333381592927698944 to 7333381592629309440. Current Snapshot: MvccSnapshot[applied={T|T < 7333381592927698944}]
I20260925 22:46:05.478127 4898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:05.478266 4898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:05.480520 4898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73b3c22af7cb4d29b0bd5aaec372ebdf: Sending DeleteTablet for 3 replicas of tablet a8c8caad60f448c69a004f36af522297
I20260925 22:46:05.481204 5288 tablet_service.cc:1514] Processing DeleteTablet for tablet a8c8caad60f448c69a004f36af522297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:05 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:39310
I20260925 22:46:05.481303 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet a8c8caad60f448c69a004f36af522297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:05 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:54260
I20260925 22:46:05.481364 5022 tablet_service.cc:1514] Processing DeleteTablet for tablet a8c8caad60f448c69a004f36af522297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:05 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:38262
I20260925 22:46:05.481747 5396 tablet_replica.cc:330] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: stopping tablet replica
I20260925 22:46:05.481894 5394 tablet_replica.cc:330] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: stopping tablet replica
I20260925 22:46:05.482048 5396 raft_consensus.cc:2229] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:05.482095 5394 raft_consensus.cc:2229] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:05.482339 5394 raft_consensus.cc:2258] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:05.482354 5396 raft_consensus.cc:2258] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:05.482637 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 4960
I20260925 22:46:05.484468 5394 ts_tablet_manager.cc:1905] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:05.485328 5396 ts_tablet_manager.cc:1905] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:05.487076 5394 ts_tablet_manager.cc:1918] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:05.487166 5394 log.cc:1196] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/a8c8caad60f448c69a004f36af522297
I20260925 22:46:05.487486 5394 ts_tablet_manager.cc:1939] T a8c8caad60f448c69a004f36af522297 P 97df6e24ebb84e039a7a8fbe3ebc2aa1: Deleting consensus metadata
I20260925 22:46:05.487977 5396 ts_tablet_manager.cc:1918] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:05.488066 5396 log.cc:1196] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/a8c8caad60f448c69a004f36af522297
I20260925 22:46:05.488356 5396 ts_tablet_manager.cc:1939] T a8c8caad60f448c69a004f36af522297 P 00e243e1b4404ffeaca6ed7733f2cd93: Deleting consensus metadata
I20260925 22:46:05.488703 4882 catalog_manager.cc:4841] TS 97df6e24ebb84e039a7a8fbe3ebc2aa1 (127.0.201.131:38491): tablet a8c8caad60f448c69a004f36af522297 (table test-table [id=5002fc6f0ce841d8910824378201f75b]) successfully deleted
W20260925 22:46:05.489039 4885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:33619 (error 108)
I20260925 22:46:05.489253 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 5093
I20260925 22:46:05.491262 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.491003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:33619 (local address 127.0.0.1:38292)
0925 22:46:05.491132 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:05.491197 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:33619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 22:46:05.491408 4885 catalog_manager.cc:4584] TS 7f74a2de8da844919f195efe63519643 (127.0.201.129:33619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8c8caad60f448c69a004f36af522297: Network error: Client connection negotiation failed: client connection to 127.0.201.129:33619: connect: Connection refused (error 111)
I20260925 22:46:05.494498 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 5226
I20260925 22:46:05.495512 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.494642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:41143 (local address 127.0.0.1:54268)
0925 22:46:05.495365 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:05.495424 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.130:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626}
I20260925 22:46:05.499665 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 4867
2026-09-25T22:46:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2430 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T22:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:05Z Disabled control of system clock
I20260925 22:46:05.530088 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42151
--webserver_interface=127.0.201.190
--webserver_port=0
--builtin_ntp_servers=127.0.201.148:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:42151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:46:05.639384 5401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:05.639644 5401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:05.639758 5401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:05.643690 5401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:05.643744 5401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:05.643759 5401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:05.643775 5401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:05.643790 5401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:05.648551 5401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:42151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:05.649562 5401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:05.650857 5401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:05.659247 5407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:05.659293 5406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:05.659307 5401 server_base.cc:1034] running on GCE node
W20260925 22:46:05.659251 5409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:05.660100 5401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:05.661188 5401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:05.662384 5401 hybrid_clock.cc:648] HybridClock initialized: now 1790376365662360 us; error 46 us; skew 500 ppm
I20260925 22:46:05.669032 5401 webserver.cc:460] Webserver started at http://127.0.201.190:40811/ using document root <none> and password file <none>
I20260925 22:46:05.669615 5401 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:05.669710 5401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:05.669951 5401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:05.672053 5401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "bda54165b28644c89e6c0fbfac19344f"
format_stamp: "Formatted at 2026-09-25 22:46:05 on dist-test-slave-vdtf"
server_key: "d1991b6dec3d4d37e8f3c8df6fbde0cc"
server_key_iv: "d5acfbdd898b226e8637803404151769"
server_key_version: "encryptionkey@0"
I20260925 22:46:05.672627 5401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "bda54165b28644c89e6c0fbfac19344f"
format_stamp: "Formatted at 2026-09-25 22:46:05 on dist-test-slave-vdtf"
server_key: "d1991b6dec3d4d37e8f3c8df6fbde0cc"
server_key_iv: "d5acfbdd898b226e8637803404151769"
server_key_version: "encryptionkey@0"
I20260925 22:46:05.676241 5401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:46:05.678503 5415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:05.679759 5401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:05.679888 5401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "bda54165b28644c89e6c0fbfac19344f"
format_stamp: "Formatted at 2026-09-25 22:46:05 on dist-test-slave-vdtf"
server_key: "d1991b6dec3d4d37e8f3c8df6fbde0cc"
server_key_iv: "d5acfbdd898b226e8637803404151769"
server_key_version: "encryptionkey@0"
I20260925 22:46:05.679996 5401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:05.684396 5401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:05.684970 5401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:05.685148 5401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:05.691656 5401 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:42151
I20260925 22:46:05.691722 5467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:42151 every 8 connection(s)
I20260925 22:46:05.692587 5401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:05.695595 5468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:05.695592 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 5401
I20260925 22:46:05.695753 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:05.696028 806 external_mini_cluster.cc:1428] Setting key fbb33147c617671dc2d9e2f54597cae6
I20260925 22:46:05.701534 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.697374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.0.201.190:42151)
0925 22:46:05.698101 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:05.698116 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:05.698150 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:46:05.698825 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:05.698842 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:05.699175 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:05.699520 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:05.700331 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:05.700335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:05.700348 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:05.700358 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:05.700409 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:05.700424 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:05.700578 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:05.700649 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:46:05.700883 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":178,"threads_started":1}
I20260925 22:46:05.702181 5468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f: Bootstrap starting.
I20260925 22:46:05.704710 5468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:05.705554 5468 log.cc:824] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:05.707584 5468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f: No bootstrap required, opened a new log
I20260925 22:46:05.711592 5468 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda54165b28644c89e6c0fbfac19344f" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42151 } }
I20260925 22:46:05.711898 5468 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:05.711985 5468 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda54165b28644c89e6c0fbfac19344f, State: Initialized, Role: FOLLOWER
I20260925 22:46:05.712427 5468 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda54165b28644c89e6c0fbfac19344f" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42151 } }
I20260925 22:46:05.712584 5468 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:05.712656 5468 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:05.712795 5468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:05.713784 5468 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda54165b28644c89e6c0fbfac19344f" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42151 } }
I20260925 22:46:05.714149 5468 leader_election.cc:302] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda54165b28644c89e6c0fbfac19344f; no voters:
I20260925 22:46:05.714483 5468 leader_election.cc:288] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:05.714592 5473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:05.714859 5473 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [term 1 LEADER]: Becoming Leader. State: Replica: bda54165b28644c89e6c0fbfac19344f, State: Running, Role: LEADER
I20260925 22:46:05.715157 5473 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda54165b28644c89e6c0fbfac19344f" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42151 } }
I20260925 22:46:05.715902 5468 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:05.716954 5475 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda54165b28644c89e6c0fbfac19344f. Latest consensus state: current_term: 1 leader_uuid: "bda54165b28644c89e6c0fbfac19344f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda54165b28644c89e6c0fbfac19344f" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42151 } } }
I20260925 22:46:05.717088 5475 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:05.717387 5474 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda54165b28644c89e6c0fbfac19344f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda54165b28644c89e6c0fbfac19344f" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42151 } } }
I20260925 22:46:05.717496 5474 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:05.717530 5482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:05.720758 5482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:05.726766 5482 catalog_manager.cc:1347] Generated new cluster ID: a821c82eb5584c869273d693cf6621e6
I20260925 22:46:05.726852 5482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:05.757248 5482 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:05.757645 5482 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:05.763657 5482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f: Generated new TSK 0
I20260925 22:46:05.764542 5482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:46:05.769313 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42151
--builtin_ntp_servers=127.0.201.148:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:46:05.883461 5492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:05.883783 5492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:05.883862 5492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:05.887957 5492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:05.888016 5492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:05.888073 5492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:05.892675 5492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:05.893731 5492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:05.895130 5492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:05.904868 5497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:05.904860 5500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:05.905028 5492 server_base.cc:1034] running on GCE node
W20260925 22:46:05.905354 5498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:05.905846 5492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:05.906687 5492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:05.907876 5492 hybrid_clock.cc:648] HybridClock initialized: now 1790376365907847 us; error 50 us; skew 500 ppm
I20260925 22:46:05.909651 5492 webserver.cc:460] Webserver started at http://127.0.201.129:38171/ using document root <none> and password file <none>
I20260925 22:46:05.910249 5492 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:05.910341 5492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:05.910554 5492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:05.912644 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "1d76f341d6d5495bba73960b280693cc"
format_stamp: "Formatted at 2026-09-25 22:46:05 on dist-test-slave-vdtf"
server_key: "588f1e74cc6a24cca9089846e372f6cd"
server_key_iv: "eb9d2a9b588e93488adf0ac49a5d48bd"
server_key_version: "encryptionkey@0"
I20260925 22:46:05.913194 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "1d76f341d6d5495bba73960b280693cc"
format_stamp: "Formatted at 2026-09-25 22:46:05 on dist-test-slave-vdtf"
server_key: "588f1e74cc6a24cca9089846e372f6cd"
server_key_iv: "eb9d2a9b588e93488adf0ac49a5d48bd"
server_key_version: "encryptionkey@0"
I20260925 22:46:05.917021 5492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:46:05.919467 5506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:05.920717 5492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:05.920852 5492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "1d76f341d6d5495bba73960b280693cc"
format_stamp: "Formatted at 2026-09-25 22:46:05 on dist-test-slave-vdtf"
server_key: "588f1e74cc6a24cca9089846e372f6cd"
server_key_iv: "eb9d2a9b588e93488adf0ac49a5d48bd"
server_key_version: "encryptionkey@0"
I20260925 22:46:05.920959 5492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:05.925450 5492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:05.926098 5492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:05.926277 5492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:05.926784 5492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:05.927651 5492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:05.927768 5492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:05.927843 5492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:05.927889 5492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:05.936344 5492 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:32797
I20260925 22:46:05.936371 5619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:32797 every 8 connection(s)
I20260925 22:46:05.937260 5492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:05.943650 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.939208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:50569 (local address 127.0.201.190:42151)
0925 22:46:05.939347 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:05.939352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:05.940548 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 22:46:05.941871 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:05.941875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:05.941920 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:05.942006 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:05.943162 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:05.943166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:05.943169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:05.943181 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:05.943202 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:05.943224 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:05.943322 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:05.943469 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 22:46:05.943514 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:46:05.944504 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:05.939606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42151 (local address 127.0.201.129:50569)
0925 22:46:05.940379 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:05.940421 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:05.941650 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:05.942183 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:05.942191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:05.942287 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:05.942904 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:05.942906 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:05.942947 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:05.943016 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:05.943314 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:05.943322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:05.943328 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:46:05.943501 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 22:46:05.943770 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":114,"threads_started":1}
I20260925 22:46:05.945514 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 5492
I20260925 22:46:05.945609 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:05.945751 5620 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42151
I20260925 22:46:05.945925 806 external_mini_cluster.cc:1428] Setting key 72a5345ee6400ee68322b26cc958dce7
I20260925 22:46:05.946025 5620 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:05.946506 5620 heartbeater.cc:511] Master 127.0.201.190:42151 requested a full tablet report, sending...
I20260925 22:46:05.947940 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42151
--builtin_ntp_servers=127.0.201.148:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:46:05.948060 5432 ts_manager.cc:194] Registered new tserver with Master: 1d76f341d6d5495bba73960b280693cc (127.0.201.129:32797)
W20260925 22:46:06.058061 5623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:06.058350 5623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:06.058391 5623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:06.062428 5623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:06.062484 5623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:06.062539 5623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:06.067248 5623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:06.068347 5623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:06.069634 5623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:06.078948 5631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:06.079046 5623 server_base.cc:1034] running on GCE node
W20260925 22:46:06.078948 5629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:06.078974 5628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:06.079537 5623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:06.080200 5623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:06.081357 5623 hybrid_clock.cc:648] HybridClock initialized: now 1790376366081330 us; error 39 us; skew 500 ppm
I20260925 22:46:06.083232 5623 webserver.cc:460] Webserver started at http://127.0.201.130:42935/ using document root <none> and password file <none>
I20260925 22:46:06.083894 5623 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:06.083985 5623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:06.084210 5623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:06.086226 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "1edb9028c432426893be2f04ac8ddc78"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "a356dc6e9e87780d9aab2b079841276c"
server_key_iv: "90c586fe2c680c12eb2707b679292b24"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.086934 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "1edb9028c432426893be2f04ac8ddc78"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "a356dc6e9e87780d9aab2b079841276c"
server_key_iv: "90c586fe2c680c12eb2707b679292b24"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.090866 5623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 22:46:06.093513 5637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:06.094801 5623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:06.094947 5623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "1edb9028c432426893be2f04ac8ddc78"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "a356dc6e9e87780d9aab2b079841276c"
server_key_iv: "90c586fe2c680c12eb2707b679292b24"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.095072 5623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:06.099665 5623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:06.100399 5623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:06.100581 5623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:06.101171 5623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:06.102039 5623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:06.102109 5623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:06.102175 5623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:06.102231 5623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:06.110582 5623 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:38939
I20260925 22:46:06.110601 5750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:38939 every 8 connection(s)
I20260925 22:46:06.111465 5623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:06.114421 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 5623
I20260925 22:46:06.114548 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:06.114892 806 external_mini_cluster.cc:1428] Setting key 897cf644b4ad5227b081012db26b0d46
I20260925 22:46:06.117084 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42151
--builtin_ntp_servers=127.0.201.148:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:46:06.118575 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.113334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:60409 (local address 127.0.201.190:42151)
0925 22:46:06.113481 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.113486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.114704 (+ 1218us) server_negotiation.cc:407] Connection header received
0925 22:46:06.116426 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.116429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.116464 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.116542 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:06.118038 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:06.118041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:06.118044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:06.118056 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:06.118077 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:06.118093 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:06.118163 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:06.118367 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 22:46:06.118401 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:46:06.119393 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.113777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42151 (local address 127.0.201.130:60409)
0925 22:46:06.114533 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:06.114588 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:06.116198 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:06.116713 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:06.116721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:06.116880 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:06.117641 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:06.117646 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:06.117748 (+ 102us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:06.117853 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:06.118180 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:06.118191 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:06.118199 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:46:06.118393 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 22:46:06.118665 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260925 22:46:06.120522 5751 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42151
I20260925 22:46:06.120750 5751 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:06.121227 5751 heartbeater.cc:511] Master 127.0.201.190:42151 requested a full tablet report, sending...
I20260925 22:46:06.122378 5432 ts_manager.cc:194] Registered new tserver with Master: 1edb9028c432426893be2f04ac8ddc78 (127.0.201.130:38939)
W20260925 22:46:06.229836 5754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:06.230084 5754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:06.230123 5754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:06.234150 5754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:06.234201 5754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:06.234292 5754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:06.239044 5754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:06.240126 5754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:06.241506 5754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:46:06.250939 5754 server_base.cc:1034] running on GCE node
W20260925 22:46:06.250861 5759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:06.250839 5760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:06.250830 5762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:06.251621 5754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:06.252240 5754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:06.253397 5754 hybrid_clock.cc:648] HybridClock initialized: now 1790376366253385 us; error 23 us; skew 500 ppm
I20260925 22:46:06.255169 5754 webserver.cc:460] Webserver started at http://127.0.201.131:33731/ using document root <none> and password file <none>
I20260925 22:46:06.255803 5754 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:06.255867 5754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:06.256106 5754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:06.258179 5754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "0d21e7a0b2e74252ab54072bf7dd8be3"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "b3f4254bd83b8bf84f8012706902d722"
server_key_iv: "328e329d57f35b8d6582dd51e97c91a8"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.258824 5754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "0d21e7a0b2e74252ab54072bf7dd8be3"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "b3f4254bd83b8bf84f8012706902d722"
server_key_iv: "328e329d57f35b8d6582dd51e97c91a8"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.262387 5754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 22:46:06.264709 5768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:06.265960 5754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:46:06.266057 5754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "0d21e7a0b2e74252ab54072bf7dd8be3"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "b3f4254bd83b8bf84f8012706902d722"
server_key_iv: "328e329d57f35b8d6582dd51e97c91a8"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.266181 5754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:06.270606 5754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:06.271162 5754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:06.271332 5754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:06.271890 5754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:06.272706 5754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:06.272749 5754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:06.272814 5754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:06.272861 5754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:06.281415 5754 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:36129
I20260925 22:46:06.281436 5881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:36129 every 8 connection(s)
I20260925 22:46:06.282248 5754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:06.283864 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 5754
I20260925 22:46:06.283979 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:46:06.284338 806 external_mini_cluster.cc:1428] Setting key 99de0f61f211a1d265aa385a4328fd08
I20260925 22:46:06.288702 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.284733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:46797 (local address 127.0.201.190:42151)
0925 22:46:06.284891 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.284894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.285591 (+ 697us) server_negotiation.cc:407] Connection header received
0925 22:46:06.287054 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.287057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.287097 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.287167 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:06.288241 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:06.288245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:06.288247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:06.288258 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:06.288277 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:06.288287 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:06.288381 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:06.288534 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 22:46:06.288577 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:46:06.289489 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.284667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42151 (local address 127.0.201.131:46797)
0925 22:46:06.285460 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:06.285513 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:06.286857 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:06.287328 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:06.287335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:06.287420 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:06.287997 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:06.287999 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:06.288038 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:06.288105 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:06.288380 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:06.288389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:06.288397 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:46:06.288565 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 22:46:06.288790 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":142,"threads_started":1}
I20260925 22:46:06.290547 5882 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42151
I20260925 22:46:06.290776 5882 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:06.291158 5882 heartbeater.cc:511] Master 127.0.201.190:42151 requested a full tablet report, sending...
I20260925 22:46:06.292104 5432 ts_manager.cc:194] Registered new tserver with Master: 0d21e7a0b2e74252ab54072bf7dd8be3 (127.0.201.131:36129)
I20260925 22:46:06.298085 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:46:06.301571 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.299920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.0.201.190:42151)
0925 22:46:06.300038 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.300043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.300231 (+ 188us) server_negotiation.cc:407] Connection header received
0925 22:46:06.300392 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.300395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.300428 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.300515 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:06.301131 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:06.301135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:06.301137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:06.301148 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:06.301168 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:06.301176 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:06.301285 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:06.301412 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 22:46:06.301451 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:46:06.304222 5432 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:46:06.307006 5432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:06.318308 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.315207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:32797 (local address 127.0.0.1:42620)
0925 22:46:06.316076 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:06.316089 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:06.316188 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:06.316755 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:06.316760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:06.316841 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:06.317436 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:06.317438 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:06.317454 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:06.317505 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:06.317976 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:06.317984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:06.317990 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:46:06.318107 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:46:06.318173 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":95,"threads_started":1}
I20260925 22:46:06.318718 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.315539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.0.201.129:32797)
0925 22:46:06.315912 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.315925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.316295 (+ 370us) server_negotiation.cc:407] Connection header received
0925 22:46:06.316396 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.316401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.316585 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.316758 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:06.317676 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:06.317683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:06.317695 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:06.317707 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:06.317760 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:06.317791 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:06.317902 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:06.318478 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 22:46:06.318558 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":89,"threads_started":1}
I20260925 22:46:06.318728 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.315264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.0.201.130:38939)
0925 22:46:06.315941 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.315961 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.316152 (+ 191us) server_negotiation.cc:407] Connection header received
0925 22:46:06.316345 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.316372 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.316573 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.316693 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:06.318153 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:06.318159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:06.318169 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:06.318181 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:06.318260 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:06.318292 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:06.318403 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:06.318505 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 22:46:06.318576 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":120,"threads_started":1}
I20260925 22:46:06.319017 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.315887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36129 (local address 127.0.0.1:34356)
0925 22:46:06.316588 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:06.316601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:06.316666 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:06.317353 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:06.317359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:06.317375 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:06.317823 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:06.317826 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:06.317837 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:06.317865 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:06.318779 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:06.318782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:06.318789 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:46:06.318872 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:46:06.318906 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":82,"threads_started":1}
I20260925 22:46:06.319482 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.316168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.0.201.131:36129)
0925 22:46:06.316916 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.316930 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.316951 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:46:06.317005 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.317012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.317200 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.317618 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:06.318185 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:06.318189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:06.318199 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:06.318207 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:06.318295 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:06.318324 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:06.319024 (+ 700us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:06.319315 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 22:46:06.319369 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":154,"threads_started":1}
I20260925 22:46:06.319769 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.315152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:38939 (local address 127.0.0.1:59060)
0925 22:46:06.315995 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:06.316042 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:06.316214 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:06.317412 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:06.317416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:06.317430 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:06.317997 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:06.317999 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:06.318015 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:06.318032 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:06.318389 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:06.318393 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:06.318395 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 22:46:06.319586 (+ 1191us) client_negotiation.cc:240] Negotiation successful
0925 22:46:06.319627 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":82,"threads_started":1}
I20260925 22:46:06.321761 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 68ec97a4e8924d1f8ae2f75ec617a4fa (DEFAULT_TABLE table=test-table [id=3ce62a774aad4a15ab9257e413b1e90d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:06.321957 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 68ec97a4e8924d1f8ae2f75ec617a4fa (DEFAULT_TABLE table=test-table [id=3ce62a774aad4a15ab9257e413b1e90d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:06.322194 5816 tablet_service.cc:1467] Processing CreateTablet for tablet 68ec97a4e8924d1f8ae2f75ec617a4fa (DEFAULT_TABLE table=test-table [id=3ce62a774aad4a15ab9257e413b1e90d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:06.322853 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ec97a4e8924d1f8ae2f75ec617a4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:06.322978 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ec97a4e8924d1f8ae2f75ec617a4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:06.323031 5816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ec97a4e8924d1f8ae2f75ec617a4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:06.328955 5899 tablet_bootstrap.cc:492] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: Bootstrap starting.
I20260925 22:46:06.330257 5901 tablet_bootstrap.cc:492] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: Bootstrap starting.
I20260925 22:46:06.330336 5900 tablet_bootstrap.cc:492] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc: Bootstrap starting.
I20260925 22:46:06.331248 5899 tablet_bootstrap.cc:654] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:06.332028 5899 log.cc:824] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:06.332341 5901 tablet_bootstrap.cc:654] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:06.332341 5900 tablet_bootstrap.cc:654] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:06.333146 5901 log.cc:824] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:06.333146 5900 log.cc:824] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:06.333945 5899 tablet_bootstrap.cc:492] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: No bootstrap required, opened a new log
I20260925 22:46:06.334146 5899 ts_tablet_manager.cc:1397] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:46:06.334990 5900 tablet_bootstrap.cc:492] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc: No bootstrap required, opened a new log
I20260925 22:46:06.335192 5900 ts_tablet_manager.cc:1397] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:46:06.335304 5901 tablet_bootstrap.cc:492] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: No bootstrap required, opened a new log
I20260925 22:46:06.335464 5901 ts_tablet_manager.cc:1397] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:46:06.338195 5899 raft_consensus.cc:348] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edb9028c432426893be2f04ac8ddc78" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38939 } } peers { permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 } } peers { permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 } }
I20260925 22:46:06.338430 5899 raft_consensus.cc:374] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:06.338481 5899 raft_consensus.cc:729] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edb9028c432426893be2f04ac8ddc78, State: Initialized, Role: FOLLOWER
I20260925 22:46:06.338912 5899 consensus_queue.cc:260] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edb9028c432426893be2f04ac8ddc78" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38939 } } peers { permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 } } peers { permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 } }
I20260925 22:46:06.339530 5751 heartbeater.cc:503] Master 127.0.201.190:42151 was elected leader, sending a full tablet report...
I20260925 22:46:06.339913 5899 ts_tablet_manager.cc:1428] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:46:06.339803 5901 raft_consensus.cc:348] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edb9028c432426893be2f04ac8ddc78" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38939 } } peers { permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 } } peers { permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 } }
I20260925 22:46:06.339803 5900 raft_consensus.cc:348] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edb9028c432426893be2f04ac8ddc78" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38939 } } peers { permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 } } peers { permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 } }
I20260925 22:46:06.340029 5901 raft_consensus.cc:374] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:06.340029 5900 raft_consensus.cc:374] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:06.340096 5900 raft_consensus.cc:729] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d76f341d6d5495bba73960b280693cc, State: Initialized, Role: FOLLOWER
I20260925 22:46:06.340101 5901 raft_consensus.cc:729] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d21e7a0b2e74252ab54072bf7dd8be3, State: Initialized, Role: FOLLOWER
I20260925 22:46:06.340535 5901 consensus_queue.cc:260] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edb9028c432426893be2f04ac8ddc78" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38939 } } peers { permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 } } peers { permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 } }
I20260925 22:46:06.340535 5900 consensus_queue.cc:260] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edb9028c432426893be2f04ac8ddc78" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38939 } } peers { permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 } } peers { permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 } }
I20260925 22:46:06.341336 5620 heartbeater.cc:503] Master 127.0.201.190:42151 was elected leader, sending a full tablet report...
I20260925 22:46:06.341462 5900 ts_tablet_manager.cc:1428] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:46:06.341460 5901 ts_tablet_manager.cc:1428] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:46:06.342371 5882 heartbeater.cc:503] Master 127.0.201.190:42151 was elected leader, sending a full tablet report...
W20260925 22:46:06.362402 5752 tablet.cc:2369] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:06.401938 5905 raft_consensus.cc:482] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:06.402122 5905 raft_consensus.cc:504] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edb9028c432426893be2f04ac8ddc78" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38939 } } peers { permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 } } peers { permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 } }
I20260925 22:46:06.403208 5905 leader_election.cc:288] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d76f341d6d5495bba73960b280693cc (127.0.201.129:32797), 0d21e7a0b2e74252ab54072bf7dd8be3 (127.0.201.131:36129)
I20260925 22:46:06.405306 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.403553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:45767 (local address 127.0.201.129:32797)
0925 22:46:06.403666 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.403730 (+ 64us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.404023 (+ 293us) server_negotiation.cc:407] Connection header received
0925 22:46:06.404097 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.404101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.404144 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.404214 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:06.404860 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:06.404864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:06.404866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:06.404879 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:06.404899 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:06.404912 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:06.404997 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:06.405138 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 22:46:06.405181 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:46:06.405323 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.403603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:40631 (local address 127.0.201.131:36129)
0925 22:46:06.403764 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.403770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.403790 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:46:06.403922 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.403924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.403958 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.404066 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:06.404820 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:06.404824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:06.404830 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:06.404842 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:06.404865 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:06.404875 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:06.404975 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:06.405145 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 22:46:06.405181 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:46:06.405321 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.403461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36129 (local address 127.0.201.130:40631)
0925 22:46:06.403661 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:06.403698 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:06.403794 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:06.404105 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:06.404108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:06.404128 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:06.404606 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:06.404607 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:06.404637 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:06.404671 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:06.405016 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:06.405020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:06.405024 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:46:06.405124 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:46:06.405165 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 22:46:06.405350 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.403490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:32797 (local address 127.0.201.130:45767)
0925 22:46:06.403928 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:06.403938 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:06.404003 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:06.404258 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:06.404263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:06.404280 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:06.404703 (+ 423us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:06.404707 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:06.404717 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:06.404741 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:06.405016 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:06.405020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:06.405024 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:46:06.405124 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:46:06.405170 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":3,"thread_start_us":165,"threads_started":1}
I20260925 22:46:06.405925 5836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ec97a4e8924d1f8ae2f75ec617a4fa" candidate_uuid: "1edb9028c432426893be2f04ac8ddc78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" is_pre_election: true
I20260925 22:46:06.405972 5574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ec97a4e8924d1f8ae2f75ec617a4fa" candidate_uuid: "1edb9028c432426893be2f04ac8ddc78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d76f341d6d5495bba73960b280693cc" is_pre_election: true
I20260925 22:46:06.406240 5836 raft_consensus.cc:2454] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1edb9028c432426893be2f04ac8ddc78 in term 0.
I20260925 22:46:06.406253 5574 raft_consensus.cc:2454] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1edb9028c432426893be2f04ac8ddc78 in term 0.
I20260925 22:46:06.406694 5638 leader_election.cc:302] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d76f341d6d5495bba73960b280693cc, 1edb9028c432426893be2f04ac8ddc78; no voters:
I20260925 22:46:06.406962 5905 raft_consensus.cc:2781] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:06.407045 5905 raft_consensus.cc:482] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:06.407086 5905 raft_consensus.cc:3037] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:06.408250 5905 raft_consensus.cc:504] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edb9028c432426893be2f04ac8ddc78" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38939 } } peers { permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 } } peers { permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 } }
I20260925 22:46:06.408589 5905 leader_election.cc:288] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [CANDIDATE]: Term 1 election: Requested vote from peers 1d76f341d6d5495bba73960b280693cc (127.0.201.129:32797), 0d21e7a0b2e74252ab54072bf7dd8be3 (127.0.201.131:36129)
I20260925 22:46:06.408890 5574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ec97a4e8924d1f8ae2f75ec617a4fa" candidate_uuid: "1edb9028c432426893be2f04ac8ddc78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d76f341d6d5495bba73960b280693cc"
I20260925 22:46:06.408968 5836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ec97a4e8924d1f8ae2f75ec617a4fa" candidate_uuid: "1edb9028c432426893be2f04ac8ddc78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3"
I20260925 22:46:06.409019 5574 raft_consensus.cc:3037] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:06.409077 5836 raft_consensus.cc:3037] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:06.410308 5836 raft_consensus.cc:2454] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1edb9028c432426893be2f04ac8ddc78 in term 1.
I20260925 22:46:06.410311 5574 raft_consensus.cc:2454] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1edb9028c432426893be2f04ac8ddc78 in term 1.
I20260925 22:46:06.410584 5640 leader_election.cc:302] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d21e7a0b2e74252ab54072bf7dd8be3, 1edb9028c432426893be2f04ac8ddc78; no voters:
I20260925 22:46:06.410753 5905 raft_consensus.cc:2781] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:06.411005 5905 raft_consensus.cc:686] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 1 LEADER]: Becoming Leader. State: Replica: 1edb9028c432426893be2f04ac8ddc78, State: Running, Role: LEADER
I20260925 22:46:06.411326 5905 consensus_queue.cc:237] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edb9028c432426893be2f04ac8ddc78" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38939 } } peers { permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 } } peers { permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 } }
I20260925 22:46:06.414578 5431 catalog_manager.cc:5495] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1edb9028c432426893be2f04ac8ddc78 (127.0.201.130). New cstate: current_term: 1 leader_uuid: "1edb9028c432426893be2f04ac8ddc78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edb9028c432426893be2f04ac8ddc78" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 } health_report { overall_health: UNKNOWN } } }
W20260925 22:46:06.438414 5621 tablet.cc:2369] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:06.439896 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.438356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.0.201.130:38939)
0925 22:46:06.438508 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.438514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.438530 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:46:06.438640 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.438644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.438717 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.438807 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:06.439455 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:06.439458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:06.439461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:06.439472 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:06.439491 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:06.439500 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:06.439598 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:06.439720 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 22:46:06.439762 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:46:06.445815 5836 raft_consensus.cc:1261] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3 [term 1 FOLLOWER]: Refusing update from remote peer 1edb9028c432426893be2f04ac8ddc78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:06.445876 5574 raft_consensus.cc:1261] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc [term 1 FOLLOWER]: Refusing update from remote peer 1edb9028c432426893be2f04ac8ddc78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:06.446568 5909 consensus_queue.cc:1035] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d21e7a0b2e74252ab54072bf7dd8be3" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:06.446799 5905 consensus_queue.cc:1035] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d76f341d6d5495bba73960b280693cc" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:06.454510 5914 mvcc.cc:205] Tried to move back new op lower bound from 7333381596954394624 to 7333381596823486464. Current Snapshot: MvccSnapshot[applied={T|T < 7333381596954394624}]
I20260925 22:46:06.462985 5432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:06.463121 5432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:06.465430 5432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bda54165b28644c89e6c0fbfac19344f: Sending DeleteTablet for 3 replicas of tablet 68ec97a4e8924d1f8ae2f75ec617a4fa
I20260925 22:46:06.466130 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ec97a4e8924d1f8ae2f75ec617a4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:06 UTC) from {username='slave'} at 127.0.0.1:42620
I20260925 22:46:06.466349 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ec97a4e8924d1f8ae2f75ec617a4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:06 UTC) from {username='slave'} at 127.0.0.1:59060
I20260925 22:46:06.466352 5816 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ec97a4e8924d1f8ae2f75ec617a4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:06 UTC) from {username='slave'} at 127.0.0.1:34356
I20260925 22:46:06.466626 5922 tablet_replica.cc:330] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc: stopping tablet replica
I20260925 22:46:06.466929 5922 raft_consensus.cc:2229] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:06.467222 5922 raft_consensus.cc:2258] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1d76f341d6d5495bba73960b280693cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:06.467283 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 5492
I20260925 22:46:06.467958 5924 tablet_replica.cc:330] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: stopping tablet replica
I20260925 22:46:06.468206 5924 raft_consensus.cc:2229] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:46:06.468559 5924 raft_consensus.cc:2258] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:06.469877 5924 ts_tablet_manager.cc:1905] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:06.472419 5923 tablet_replica.cc:330] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: stopping tablet replica
I20260925 22:46:06.472590 5924 ts_tablet_manager.cc:1918] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:06.472683 5924 log.cc:1196] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/68ec97a4e8924d1f8ae2f75ec617a4fa
I20260925 22:46:06.472885 5923 raft_consensus.cc:2229] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:06.473009 5924 ts_tablet_manager.cc:1939] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 1edb9028c432426893be2f04ac8ddc78: Deleting consensus metadata
I20260925 22:46:06.473284 5923 raft_consensus.cc:2258] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:06.474196 5418 catalog_manager.cc:4841] TS 1edb9028c432426893be2f04ac8ddc78 (127.0.201.130:38939): tablet 68ec97a4e8924d1f8ae2f75ec617a4fa (table test-table [id=3ce62a774aad4a15ab9257e413b1e90d]) successfully deleted
I20260925 22:46:06.474804 5923 ts_tablet_manager.cc:1905] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:46:06.476002 5416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:32797 (error 108)
I20260925 22:46:06.476213 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 5623
I20260925 22:46:06.477313 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.476815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:32797 (local address 127.0.0.1:42622)
0925 22:46:06.477193 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:06.477249 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312}
W20260925 22:46:06.477833 5416 catalog_manager.cc:4584] TS 1d76f341d6d5495bba73960b280693cc (127.0.201.129:32797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68ec97a4e8924d1f8ae2f75ec617a4fa: Network error: Client connection negotiation failed: client connection to 127.0.201.129:32797: connect: Connection refused (error 111)
I20260925 22:46:06.478070 5923 ts_tablet_manager.cc:1918] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:06.478168 5923 log.cc:1196] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/68ec97a4e8924d1f8ae2f75ec617a4fa
I20260925 22:46:06.478485 5923 ts_tablet_manager.cc:1939] T 68ec97a4e8924d1f8ae2f75ec617a4fa P 0d21e7a0b2e74252ab54072bf7dd8be3: Deleting consensus metadata
I20260925 22:46:06.479542 5418 catalog_manager.cc:4841] TS 0d21e7a0b2e74252ab54072bf7dd8be3 (127.0.201.131:36129): tablet 68ec97a4e8924d1f8ae2f75ec617a4fa (table test-table [id=3ce62a774aad4a15ab9257e413b1e90d]) successfully deleted
I20260925 22:46:06.482587 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 5754
I20260925 22:46:06.488081 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 5401
2026-09-25T22:46:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (986 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 22:46:06.509070 806 mini_oidc.cc:152] Starting JWKS server
I20260925 22:46:06.510627 806 webserver.cc:460] Webserver started at https://127.0.0.1:43935/ using document root <none> and password file <none>
I20260925 22:46:06.510818 806 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:46:06.511086 806 webserver.cc:460] Webserver started at http://127.0.0.1:46441/ using document root <none> and password file <none>
2026-09-25T22:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:06Z Disabled control of system clock
I20260925 22:46:06.519495 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:32797
--webserver_interface=127.0.201.190
--webserver_port=0
--builtin_ntp_servers=127.0.201.148:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:32797
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43935/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/testchainca.cert with env {}
W20260925 22:46:06.627947 5931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:06.628213 5931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:06.628252 5931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:06.632359 5931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:06.632416 5931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:06.632431 5931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:06.632448 5931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:06.632489 5931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:46:06.632521 5931 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:46:06.632539 5931 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43935/jwks.json
I20260925 22:46:06.638497 5931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:32797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:32797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43935/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:06.639559 5931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:06.640622 5931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:06.649138 5936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:06.649144 5937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:06.649286 5931 server_base.cc:1034] running on GCE node
W20260925 22:46:06.649217 5939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:06.649963 5931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:06.651085 5931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:06.652275 5931 hybrid_clock.cc:648] HybridClock initialized: now 1790376366652244 us; error 41 us; skew 500 ppm
I20260925 22:46:06.654036 5931 webserver.cc:460] Webserver started at http://127.0.201.190:43447/ using document root <none> and password file <none>
I20260925 22:46:06.654623 5931 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:06.654714 5931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:06.654946 5931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:06.656952 5931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "a45022130320417cae724831d9bb7cf5"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "56f8ca19234d739fec2604ead45faf12"
server_key_iv: "85f8429005e8ca944b810224e96f7b6f"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.657516 5931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "a45022130320417cae724831d9bb7cf5"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "56f8ca19234d739fec2604ead45faf12"
server_key_iv: "85f8429005e8ca944b810224e96f7b6f"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.661151 5931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:06.663493 5945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:06.664855 5931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:06.665009 5931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "a45022130320417cae724831d9bb7cf5"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "56f8ca19234d739fec2604ead45faf12"
server_key_iv: "85f8429005e8ca944b810224e96f7b6f"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.665120 5931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:06.677876 5931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:06.678481 5931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:06.678669 5931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:06.685386 5931 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:32797
I20260925 22:46:06.685395 5997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:32797 every 8 connection(s)
I20260925 22:46:06.686256 5931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:06.689278 5998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:06.694371 5998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5: Bootstrap starting.
I20260925 22:46:06.694518 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 5931
I20260925 22:46:06.694646 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:06.694895 806 external_mini_cluster.cc:1428] Setting key 7cd2e033096759b5c60c2ec0fe758538
I20260925 22:46:06.697003 5998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:06.697846 5998 log.cc:824] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:06.699858 5998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5: No bootstrap required, opened a new log
I20260925 22:46:06.703435 5998 raft_consensus.cc:348] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45022130320417cae724831d9bb7cf5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 32797 } }
I20260925 22:46:06.703672 5998 raft_consensus.cc:374] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:06.703830 5998 raft_consensus.cc:729] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a45022130320417cae724831d9bb7cf5, State: Initialized, Role: FOLLOWER
I20260925 22:46:06.704267 5998 consensus_queue.cc:260] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45022130320417cae724831d9bb7cf5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 32797 } }
I20260925 22:46:06.704417 5998 raft_consensus.cc:388] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:06.704448 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.696222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.0.201.190:32797)
0925 22:46:06.696862 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.696882 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.696926 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:46:06.697760 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.697781 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.698260 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.698606 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:06.699009 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:06.700043 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:06.700939 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:06.701404 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:06.701570 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:06.702469 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:06.702474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:06.702487 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:06.702872 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:06.702930 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:06.702951 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:06.703062 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:06.703211 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 22:46:06.703487 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":107,"threads_started":1}
I20260925 22:46:06.704502 5998 raft_consensus.cc:482] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:06.704623 5998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:06.705643 5998 raft_consensus.cc:504] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45022130320417cae724831d9bb7cf5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 32797 } }
I20260925 22:46:06.705976 5998 leader_election.cc:302] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a45022130320417cae724831d9bb7cf5; no voters:
I20260925 22:46:06.706295 5998 leader_election.cc:288] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:06.706379 6003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:06.706557 6003 raft_consensus.cc:686] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [term 1 LEADER]: Becoming Leader. State: Replica: a45022130320417cae724831d9bb7cf5, State: Running, Role: LEADER
I20260925 22:46:06.706849 6003 consensus_queue.cc:237] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45022130320417cae724831d9bb7cf5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 32797 } }
I20260925 22:46:06.707267 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:46:06.707508 5998 sys_catalog.cc:564] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:06.708603 6005 sys_catalog.cc:455] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a45022130320417cae724831d9bb7cf5. Latest consensus state: current_term: 1 leader_uuid: "a45022130320417cae724831d9bb7cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45022130320417cae724831d9bb7cf5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 32797 } } }
I20260925 22:46:06.708729 6005 sys_catalog.cc:458] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:06.709041 6004 sys_catalog.cc:455] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a45022130320417cae724831d9bb7cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45022130320417cae724831d9bb7cf5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 32797 } } }
I20260925 22:46:06.709142 6004 sys_catalog.cc:458] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:06.709385 6012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:06.712385 6012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:06.717056 6012 catalog_manager.cc:1347] Generated new cluster ID: 6cf352dac9eb491d9dc121c4c9e1da90
I20260925 22:46:06.717115 6012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:06.743467 6012 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:06.744431 6012 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:06.749302 6012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a45022130320417cae724831d9bb7cf5: Generated new TSK 0
I20260925 22:46:06.750094 6012 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:46:06.784313 6028 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 22:46:06.784425 6028 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:46:06.785007 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:06.768003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.0.201.190:32797)
0925 22:46:06.768135 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:06.768140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:06.768307 (+ 167us) server_negotiation.cc:407] Connection header received
0925 22:46:06.768511 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:06.768515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:06.768571 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:06.768684 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:46:06.768986 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:06.769852 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:06.770698 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:06.771054 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:06.771163 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:06.771473 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:46:06.784615 (+ 13142us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:46:06.784866 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 22:46:06.785085 6001 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 22:46:06.786077 6027 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:06.767912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:32797 (local address 127.0.0.1:44018)
0925 22:46:06.768229 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:06.768242 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:06.768352 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:06.768728 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:06.768731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:06.768739 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:46:06.768879 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:06.768884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:06.769971 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:06.769974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:06.770577 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:06.770583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:06.771178 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:06.771181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:06.771246 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:06.771285 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:46:06.784805 (+ 13520us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:46:06.784997 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.201.190:32797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.201.190:32797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 22:46:06.786783 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 5931
2026-09-25T22:46:06Z chronyd exiting
I20260925 22:46:06.824095 806 test_util.cc:175] -----------------------------------------------
I20260925 22:46:06.824200 806 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376341615954-806-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 22:46:06.825287 806 mini_oidc.cc:152] Starting JWKS server
I20260925 22:46:06.826150 806 webserver.cc:460] Webserver started at https://127.0.0.1:44649/ using document root <none> and password file <none>
I20260925 22:46:06.826279 806 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:46:06.826572 806 webserver.cc:460] Webserver started at http://127.0.0.1:42145/ using document root <none> and password file <none>
2026-09-25T22:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:06Z Disabled control of system clock
I20260925 22:46:06.835027 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40013
--webserver_interface=127.0.201.190
--webserver_port=0
--builtin_ntp_servers=127.0.201.148:46443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:40013
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44649/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/testchainca.cert with env {}
W20260925 22:46:06.942731 6034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:06.943040 6034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:06.943126 6034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:06.947103 6034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:06.947201 6034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:06.947232 6034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:06.947270 6034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:06.947302 6034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:46:06.947352 6034 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:46:06.947386 6034 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44649/jwks.json
I20260925 22:46:06.952845 6034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:46443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:40013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44649/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:06.953943 6034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:06.954985 6034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:06.963196 6042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:06.963302 6039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:06.963305 6040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:06.963492 6034 server_base.cc:1034] running on GCE node
I20260925 22:46:06.964120 6034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:06.965200 6034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:06.966360 6034 hybrid_clock.cc:648] HybridClock initialized: now 1790376366966346 us; error 26 us; skew 500 ppm
I20260925 22:46:06.968183 6034 webserver.cc:460] Webserver started at http://127.0.201.190:38333/ using document root <none> and password file <none>
I20260925 22:46:06.968731 6034 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:06.968791 6034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:06.968997 6034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:06.971060 6034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "83d8aaa3b24841f4be3b40fc045cf259"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "963e00d332395a57abcd4720edecbeef"
server_key_iv: "1339356cc980e2acfad281b3104a3f83"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.971654 6034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "83d8aaa3b24841f4be3b40fc045cf259"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "963e00d332395a57abcd4720edecbeef"
server_key_iv: "1339356cc980e2acfad281b3104a3f83"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.975175 6034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:46:06.977537 6048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:06.978679 6034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:06.978798 6034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "83d8aaa3b24841f4be3b40fc045cf259"
format_stamp: "Formatted at 2026-09-25 22:46:06 on dist-test-slave-vdtf"
server_key: "963e00d332395a57abcd4720edecbeef"
server_key_iv: "1339356cc980e2acfad281b3104a3f83"
server_key_version: "encryptionkey@0"
I20260925 22:46:06.978906 6034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:06.993665 6034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:06.994273 6034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:06.994457 6034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:07.001643 6034 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:40013
I20260925 22:46:07.001682 6100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:40013 every 8 connection(s)
I20260925 22:46:07.002571 6034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:07.005657 6101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:07.010442 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 6034
I20260925 22:46:07.010584 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:07.010861 806 external_mini_cluster.cc:1428] Setting key bc142af91813707d81e76d0ac7c694c5
I20260925 22:46:07.011109 6101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259: Bootstrap starting.
I20260925 22:46:07.013495 6101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:07.014272 6101 log.cc:824] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:07.016369 6101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259: No bootstrap required, opened a new log
I20260925 22:46:07.019865 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:07.012244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.0.201.190:40013)
0925 22:46:07.012850 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:07.012864 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:07.012899 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:46:07.013492 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:07.013507 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:07.013941 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:07.014221 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:07.014602 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.015519 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.016754 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.017192 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.017305 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:07.018123 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:07.018127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:07.018139 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:07.018497 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:07.018573 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:07.018592 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:07.018667 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:07.018849 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 22:46:07.019123 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":129,"threads_started":1}
I20260925 22:46:07.020396 6101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d8aaa3b24841f4be3b40fc045cf259" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40013 } }
I20260925 22:46:07.020599 6101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:07.020656 6101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d8aaa3b24841f4be3b40fc045cf259, State: Initialized, Role: FOLLOWER
I20260925 22:46:07.021080 6101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d8aaa3b24841f4be3b40fc045cf259" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40013 } }
I20260925 22:46:07.021245 6101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:07.021324 6101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:07.021447 6101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:07.022498 6101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d8aaa3b24841f4be3b40fc045cf259" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40013 } }
I20260925 22:46:07.022652 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:46:07.022879 6101 leader_election.cc:302] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83d8aaa3b24841f4be3b40fc045cf259; no voters:
I20260925 22:46:07.023217 6101 leader_election.cc:288] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:07.023306 6106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:07.023506 6106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [term 1 LEADER]: Becoming Leader. State: Replica: 83d8aaa3b24841f4be3b40fc045cf259, State: Running, Role: LEADER
I20260925 22:46:07.023847 6106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d8aaa3b24841f4be3b40fc045cf259" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40013 } }
I20260925 22:46:07.024483 6101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:07.025498 6108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d8aaa3b24841f4be3b40fc045cf259. Latest consensus state: current_term: 1 leader_uuid: "83d8aaa3b24841f4be3b40fc045cf259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d8aaa3b24841f4be3b40fc045cf259" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40013 } } }
I20260925 22:46:07.025624 6108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:07.025507 6107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83d8aaa3b24841f4be3b40fc045cf259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d8aaa3b24841f4be3b40fc045cf259" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40013 } } }
I20260925 22:46:07.026119 6107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259 [sys.catalog]: This master's current role is: LEADER
W20260925 22:46:07.029044 6121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:46:07.029111 6121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:46:07.029204 6115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:07.029841 6115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:07.034924 6115 catalog_manager.cc:1347] Generated new cluster ID: b191dc7f807c4344bd1df979e311e37f
I20260925 22:46:07.034986 6115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:07.047034 6115 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:07.048200 6115 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:07.055874 6115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83d8aaa3b24841f4be3b40fc045cf259: Generated new TSK 0
I20260925 22:46:07.056612 6115 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:46:07.079764 6131 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 22:46:07.079859 6131 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:46:07.080454 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:07.064704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.0.201.190:40013)
0925 22:46:07.064831 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:07.064835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:07.064996 (+ 161us) server_negotiation.cc:407] Connection header received
0925 22:46:07.065118 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:07.065121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:07.065174 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:07.065287 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:46:07.065487 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.066288 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.067172 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.067525 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.067629 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:07.067964 (+ 335us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:46:07.080040 (+ 12076us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:46:07.080297 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33134: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:46:07.080518 6104 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33134: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 22:46:07.080579 6130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:07.064574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40013 (local address 127.0.0.1:33134)
0925 22:46:07.064913 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:07.064926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:07.065017 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:07.065292 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:07.065294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:07.065300 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:46:07.065389 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:07.065394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.066435 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.066438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:07.067050 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:07.067057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.067632 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.067634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:07.067777 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:07.067802 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:46:07.080244 (+ 12442us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:46:07.080407 (+ 163us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.201.190:40013: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
I20260925 22:46:07.081292 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 6034
2026-09-25T22:46:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (294 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 22:46:07.120299 806 mini_oidc.cc:152] Starting JWKS server
I20260925 22:46:07.121158 806 webserver.cc:460] Webserver started at https://127.0.0.1:42301/ using document root <none> and password file <none>
I20260925 22:46:07.121285 806 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:46:07.121548 806 webserver.cc:460] Webserver started at http://127.0.0.1:37487/ using document root <none> and password file <none>
2026-09-25T22:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:07Z Disabled control of system clock
I20260925 22:46:07.130203 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:35763
--webserver_interface=127.0.201.190
--webserver_port=0
--builtin_ntp_servers=127.0.201.148:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:35763
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42301/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:46:07.237434 6137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:07.237679 6137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:07.237742 6137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:07.241725 6137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:07.241791 6137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:07.241807 6137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:07.241868 6137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:07.241896 6137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:46:07.241951 6137 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:46:07.241990 6137 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42301/jwks.json
I20260925 22:46:07.246788 6137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:35763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:35763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42301/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:07.247854 6137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:07.249208 6137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:07.257494 6145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:07.257488 6142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:07.257488 6143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:07.257936 6137 server_base.cc:1034] running on GCE node
I20260925 22:46:07.258558 6137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:07.259648 6137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:07.260918 6137 hybrid_clock.cc:648] HybridClock initialized: now 1790376367260854 us; error 82 us; skew 500 ppm
I20260925 22:46:07.262746 6137 webserver.cc:460] Webserver started at http://127.0.201.190:34657/ using document root <none> and password file <none>
I20260925 22:46:07.263314 6137 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:07.263405 6137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:07.263634 6137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:07.265668 6137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3"
format_stamp: "Formatted at 2026-09-25 22:46:07 on dist-test-slave-vdtf"
server_key: "f01e63d4fb69ad01369158b55ad46ee6"
server_key_iv: "c21c315d5993395e633e8fe4dd8c8632"
server_key_version: "encryptionkey@0"
I20260925 22:46:07.266216 6137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3"
format_stamp: "Formatted at 2026-09-25 22:46:07 on dist-test-slave-vdtf"
server_key: "f01e63d4fb69ad01369158b55ad46ee6"
server_key_iv: "c21c315d5993395e633e8fe4dd8c8632"
server_key_version: "encryptionkey@0"
I20260925 22:46:07.269979 6137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260925 22:46:07.272243 6151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:07.273408 6137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:07.273536 6137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3"
format_stamp: "Formatted at 2026-09-25 22:46:07 on dist-test-slave-vdtf"
server_key: "f01e63d4fb69ad01369158b55ad46ee6"
server_key_iv: "c21c315d5993395e633e8fe4dd8c8632"
server_key_version: "encryptionkey@0"
I20260925 22:46:07.273636 6137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:07.304785 6137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:07.305516 6137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:07.305714 6137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:07.312515 6137 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:35763
I20260925 22:46:07.312542 6203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:35763 every 8 connection(s)
I20260925 22:46:07.313459 6137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:07.315528 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 6137
I20260925 22:46:07.315658 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:07.315981 806 external_mini_cluster.cc:1428] Setting key da3449fed143872b1cbb729f70fe44cc
I20260925 22:46:07.316747 6204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:07.322491 6204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3: Bootstrap starting.
I20260925 22:46:07.324769 6204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:07.324791 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:07.317271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.0.201.190:35763)
0925 22:46:07.317911 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:07.317926 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:07.317961 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:46:07.318601 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:07.318616 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:07.318986 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:07.319328 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:07.319788 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.320736 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.321639 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.322108 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.322277 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:07.323124 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:07.323129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:07.323140 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:07.323514 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:07.323564 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:07.323583 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:07.323710 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:07.323877 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 22:46:07.324121 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260925 22:46:07.325546 6204 log.cc:824] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:07.327373 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:46:07.327513 6204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3: No bootstrap required, opened a new log
I20260925 22:46:07.330993 6204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35763 } }
I20260925 22:46:07.331219 6204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:07.331300 6204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c988f4b8ae449f3ad6ab1bdeb6c00b3, State: Initialized, Role: FOLLOWER
I20260925 22:46:07.331779 6204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35763 } }
I20260925 22:46:07.331940 6204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:07.332008 6204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:07.332103 6204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:07.332918 6204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35763 } }
I20260925 22:46:07.333235 6204 leader_election.cc:302] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c988f4b8ae449f3ad6ab1bdeb6c00b3; no voters:
I20260925 22:46:07.333547 6204 leader_election.cc:288] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:07.333616 6209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:07.333811 6209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [term 1 LEADER]: Becoming Leader. State: Replica: 5c988f4b8ae449f3ad6ab1bdeb6c00b3, State: Running, Role: LEADER
I20260925 22:46:07.334077 6209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35763 } }
I20260925 22:46:07.334748 6204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:07.335580 6210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35763 } } }
I20260925 22:46:07.335726 6210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:07.336061 6218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:07.336088 6211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c988f4b8ae449f3ad6ab1bdeb6c00b3. Latest consensus state: current_term: 1 leader_uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c988f4b8ae449f3ad6ab1bdeb6c00b3" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35763 } } }
I20260925 22:46:07.336241 6211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:07.339293 6218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:07.344807 6218 catalog_manager.cc:1347] Generated new cluster ID: 7202d512ec1840aaab3c0aabf479a305
I20260925 22:46:07.344880 6218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:07.364524 6218 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:07.365474 6218 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:07.374569 6218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c988f4b8ae449f3ad6ab1bdeb6c00b3: Generated new TSK 0
I20260925 22:46:07.375319 6218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:46:07.390884 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:07.385756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.0.201.190:35763)
0925 22:46:07.385897 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:07.385901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:07.386099 (+ 198us) server_negotiation.cc:407] Connection header received
0925 22:46:07.386262 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:07.386264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:07.386313 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:07.386403 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:46:07.386647 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.387176 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.388453 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.388660 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.388740 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:07.389070 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:46:07.390524 (+ 1454us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42301/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:46:07.390746 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42301/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 22:46:07.390962 6207 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42301/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 22:46:07.392822 6233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:07.385699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:35763 (local address 127.0.0.1:33286)
0925 22:46:07.386005 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:07.386020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:07.386134 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:07.386425 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:07.386428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:07.386436 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:46:07.386544 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:07.386550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.387290 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.387293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:07.388302 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:07.388312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.388778 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.388782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:07.388878 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:07.388907 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:46:07.390688 (+ 1781us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42301/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:46:07.392619 (+ 1931us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.201.190:35763: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42301/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":1686,"thread_start_us":120,"threads_started":1}
I20260925 22:46:07.393407 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 6137
E20260925 22:46:07.402129 6234 webserver.cc:533] Webserver: sslize failed:
E20260925 22:46:07.402226 6234 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T22:46:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T22:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:07Z Disabled control of system clock
I20260925 22:46:07.440783 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:45481
--webserver_interface=127.0.201.190
--webserver_port=0
--builtin_ntp_servers=127.0.201.148:41491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:45481
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/testchainca.cert with env {}
W20260925 22:46:07.548923 6238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:07.549189 6238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:07.549228 6238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:07.553239 6238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:07.553297 6238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:07.553318 6238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:07.553335 6238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:07.553351 6238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:46:07.553411 6238 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:46:07.553431 6238 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 22:46:07.558840 6238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:45481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:45481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:07.559950 6238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:07.561056 6238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:07.569701 6246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:07.569697 6243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:07.569911 6238 server_base.cc:1034] running on GCE node
W20260925 22:46:07.569826 6244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:07.570652 6238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:07.571775 6238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:07.572939 6238 hybrid_clock.cc:648] HybridClock initialized: now 1790376367572911 us; error 40 us; skew 500 ppm
I20260925 22:46:07.574741 6238 webserver.cc:460] Webserver started at http://127.0.201.190:34367/ using document root <none> and password file <none>
I20260925 22:46:07.575344 6238 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:07.575407 6238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:07.575652 6238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:07.577715 6238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885"
format_stamp: "Formatted at 2026-09-25 22:46:07 on dist-test-slave-vdtf"
server_key: "87f529ee2fb9865f8eb1d7809ef92196"
server_key_iv: "fef58b4b0394635c1ed893fe23a6d848"
server_key_version: "encryptionkey@0"
I20260925 22:46:07.578292 6238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885"
format_stamp: "Formatted at 2026-09-25 22:46:07 on dist-test-slave-vdtf"
server_key: "87f529ee2fb9865f8eb1d7809ef92196"
server_key_iv: "fef58b4b0394635c1ed893fe23a6d848"
server_key_version: "encryptionkey@0"
I20260925 22:46:07.581933 6238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:07.584259 6252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:07.585477 6238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:07.585597 6238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885"
format_stamp: "Formatted at 2026-09-25 22:46:07 on dist-test-slave-vdtf"
server_key: "87f529ee2fb9865f8eb1d7809ef92196"
server_key_iv: "fef58b4b0394635c1ed893fe23a6d848"
server_key_version: "encryptionkey@0"
I20260925 22:46:07.585700 6238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:07.617743 6238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:07.618482 6238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:07.618690 6238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:07.625820 6304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:45481 every 8 connection(s)
I20260925 22:46:07.625823 6238 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:45481
I20260925 22:46:07.626839 6238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:07.629963 6305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:07.635452 6305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885: Bootstrap starting.
I20260925 22:46:07.636629 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 6238
I20260925 22:46:07.636754 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:07.637049 806 external_mini_cluster.cc:1428] Setting key addf03c40593ac75a49bfdaab4d30bbc
I20260925 22:46:07.638394 6305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:07.639124 6305 log.cc:824] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:07.641268 6305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885: No bootstrap required, opened a new log
I20260925 22:46:07.645220 6305 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45481 } }
I20260925 22:46:07.645511 6305 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:07.645589 6305 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb5d133dfbb4b94b4c66aa7a1d69885, State: Initialized, Role: FOLLOWER
I20260925 22:46:07.646117 6305 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45481 } }
I20260925 22:46:07.646248 6305 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:07.646340 6305 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:07.646458 6305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:07.646550 6308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:07.638086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.0.201.190:45481)
0925 22:46:07.638801 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:07.638816 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:07.638853 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:46:07.639570 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:07.639593 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:07.640058 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:07.640459 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:07.640877 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.641890 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.642802 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.643311 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.643490 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:07.644513 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:07.644518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:07.644532 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:07.644947 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:07.645003 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:07.645023 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:07.645140 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:07.645301 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 22:46:07.645633 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":149,"threads_started":1}
I20260925 22:46:07.647490 6305 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45481 } }
I20260925 22:46:07.647907 6305 leader_election.cc:302] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdb5d133dfbb4b94b4c66aa7a1d69885; no voters:
I20260925 22:46:07.648249 6305 leader_election.cc:288] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:07.648353 6310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:07.648576 6310 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [term 1 LEADER]: Becoming Leader. State: Replica: fdb5d133dfbb4b94b4c66aa7a1d69885, State: Running, Role: LEADER
I20260925 22:46:07.648883 6310 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45481 } }
I20260925 22:46:07.649590 6305 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:07.649694 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:46:07.650683 6312 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdb5d133dfbb4b94b4c66aa7a1d69885. Latest consensus state: current_term: 1 leader_uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45481 } } }
I20260925 22:46:07.650679 6311 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb5d133dfbb4b94b4c66aa7a1d69885" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45481 } } }
I20260925 22:46:07.650835 6312 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:07.650873 6311 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:07.651440 6319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:07.654238 6319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:07.659132 6319 catalog_manager.cc:1347] Generated new cluster ID: 36ccc68352ac49a2a5ea4a60110f1359
I20260925 22:46:07.659192 6319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:07.671317 6319 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:07.672614 6319 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:07.681563 6319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdb5d133dfbb4b94b4c66aa7a1d69885: Generated new TSK 0
I20260925 22:46:07.682404 6319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:46:07.741098 6308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:07.695395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.0.201.190:45481)
0925 22:46:07.695533 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:07.695536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:07.695654 (+ 118us) server_negotiation.cc:407] Connection header received
0925 22:46:07.695845 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:07.695847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:07.695891 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:07.695979 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:46:07.696223 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.696755 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.698033 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.698259 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.698345 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:07.698657 (+ 312us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:46:07.740703 (+ 42046us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:46:07.740978 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59972: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 22:46:07.741204 6308 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59972: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 22:46:07.741353 6334 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:07.695249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45481 (local address 127.0.0.1:59972)
0925 22:46:07.695559 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:07.695574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:07.695711 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:07.696003 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:07.696005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:07.696014 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:46:07.696124 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:07.696130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.696859 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.696863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:07.697907 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:07.697919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:07.698386 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:07.698390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:07.698480 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:07.698508 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:46:07.740946 (+ 42438us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:46:07.741154 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.201.190:45481: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260925 22:46:07.742070 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 6238
2026-09-25T22:46:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (328 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:46:08 dist-test-slave-vdtf krb5kdc[6346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:46:08 dist-test-slave-vdtf krb5kdc[6346](info): set up 2 sockets
Sep 25 22:46:08 dist-test-slave-vdtf krb5kdc[6346](info): commencing operation
krb5kdc: starting...
W20260925 22:46:09.012065 806 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:46:09 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:46:09.130439 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42519
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:42519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:09.239414 6359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:09.239751 6359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:09.239862 6359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:09.243849 6359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:09.243934 6359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:09.243973 6359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:09.244009 6359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:09.244056 6359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:09.249387 6359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:42519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:09.250432 6359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:09.251467 6359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:09.259722 6364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:09.259840 6359 server_base.cc:1034] running on GCE node
W20260925 22:46:09.259740 6367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:09.260062 6365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:09.260648 6359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:09.261677 6359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:09.262831 6359 hybrid_clock.cc:648] HybridClock initialized: now 1790376369262821 us; error 24 us; skew 500 ppm
Sep 25 22:46:09 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:09.265628 6359 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:46:09.266486 6359 webserver.cc:460] Webserver started at http://127.0.201.190:36805/ using document root <none> and password file <none>
I20260925 22:46:09.267019 6359 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:09.267077 6359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:09.267295 6359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:09.269345 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "dd2253dd88104e958b68cbbb381fa22c"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "77d26ba1c7a1e11e3a666491f4b46253"
server_key_iv: "4549d6c37c562b45c7e86c340aefa351"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.269918 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "dd2253dd88104e958b68cbbb381fa22c"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "77d26ba1c7a1e11e3a666491f4b46253"
server_key_iv: "4549d6c37c562b45c7e86c340aefa351"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.273584 6359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:46:09.276443 6374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:09.277604 6359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:09.277726 6359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "dd2253dd88104e958b68cbbb381fa22c"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "77d26ba1c7a1e11e3a666491f4b46253"
server_key_iv: "4549d6c37c562b45c7e86c340aefa351"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.277833 6359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:09.288841 6359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:09.291507 6359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:09.291774 6359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:09.298525 6359 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:42519
I20260925 22:46:09.298830 6426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:42519 every 8 connection(s)
I20260925 22:46:09.299464 6359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:09.302145 6427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:09.306370 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 6359
I20260925 22:46:09.306466 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:09.306752 806 external_mini_cluster.cc:1428] Setting key 5df8418bed8bcb34104c4ebbde9e4879
I20260925 22:46:09.307835 6427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c: Bootstrap starting.
I20260925 22:46:09.310019 6427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:09.310894 6427 log.cc:824] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:09.312891 6427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c: No bootstrap required, opened a new log
Sep 25 22:46:09 dist-test-slave-vdtf krb5kdc[6346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:09.317065 6427 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2253dd88104e958b68cbbb381fa22c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42519 } }
I20260925 22:46:09.317312 6427 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:09.317375 6427 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2253dd88104e958b68cbbb381fa22c, State: Initialized, Role: FOLLOWER
I20260925 22:46:09.317818 6427 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2253dd88104e958b68cbbb381fa22c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42519 } }
I20260925 22:46:09.317965 6427 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:09.318048 6427 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:09.318161 6427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:09.318977 6427 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2253dd88104e958b68cbbb381fa22c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42519 } }
I20260925 22:46:09.319303 6427 leader_election.cc:302] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd2253dd88104e958b68cbbb381fa22c; no voters:
I20260925 22:46:09.319594 6427 leader_election.cc:288] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:09.319713 6432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:09.319936 6432 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [term 1 LEADER]: Becoming Leader. State: Replica: dd2253dd88104e958b68cbbb381fa22c, State: Running, Role: LEADER
I20260925 22:46:09.320250 6432 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2253dd88104e958b68cbbb381fa22c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42519 } }
I20260925 22:46:09.320807 6427 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:09.321410 6432 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd2253dd88104e958b68cbbb381fa22c. Latest consensus state: current_term: 1 leader_uuid: "dd2253dd88104e958b68cbbb381fa22c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2253dd88104e958b68cbbb381fa22c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42519 } } }
I20260925 22:46:09.321521 6432 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:09.321955 6433 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd2253dd88104e958b68cbbb381fa22c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2253dd88104e958b68cbbb381fa22c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42519 } } }
I20260925 22:46:09.322049 6433 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:09.322216 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:09.308188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.0.201.190:42519)
0925 22:46:09.308573 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:09.308588 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:09.308622 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:46:09.309216 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:09.309231 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:09.309558 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:09.309826 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:09.310765 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:09.312033 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:09.313137 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:09.313579 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:09.313688 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:09.316509 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:09.316514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:09.316525 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:09.316550 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:09.318443 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:09.318861 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:09.318867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:09.318872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:09.318993 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:09.319273 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:09.319276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:09.319278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:09.319841 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:09.320697 (+ 856us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:09.320848 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:46:09.321136 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":129,"threads_started":1}
W20260925 22:46:09.325829 6448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:46:09.325914 6448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:46:09.325990 6447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:09.326639 6447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:09.332172 6447 catalog_manager.cc:1347] Generated new cluster ID: ce774f907f3c4baaaf2d9fbe00a3bb9a
I20260925 22:46:09.332231 6447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:09.342975 6447 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:09.343968 6447 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:09.350136 6447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd2253dd88104e958b68cbbb381fa22c: Generated new TSK 0
I20260925 22:46:09.350754 6447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:46:09.382601 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42519
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:09.491308 6453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:09.491537 6453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:09.491576 6453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:09.495430 6453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:09.495476 6453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:09.495530 6453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:09.500505 6453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:09.501385 6453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:09.502266 6453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:09.511673 6461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:09.511695 6459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:09.511684 6458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:09.511745 6453 server_base.cc:1034] running on GCE node
I20260925 22:46:09.512231 6453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:09.512821 6453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:09.514001 6453 hybrid_clock.cc:648] HybridClock initialized: now 1790376369513987 us; error 26 us; skew 500 ppm
Sep 25 22:46:09 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:09.516711 6453 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:46:09.517503 6453 webserver.cc:460] Webserver started at http://127.0.201.129:34723/ using document root <none> and password file <none>
I20260925 22:46:09.517994 6453 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:09.518049 6453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:09.518199 6453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:09.520098 6453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "7b543de346274816a1547de7a70242c0"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "0a0169f359eff97c8293b40644401d8f"
server_key_iv: "1386c0e0a236bae6335eb1d3fae77a99"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.520598 6453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "7b543de346274816a1547de7a70242c0"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "0a0169f359eff97c8293b40644401d8f"
server_key_iv: "1386c0e0a236bae6335eb1d3fae77a99"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.524125 6453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:46:09.526332 6468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:09.527510 6453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:09.527645 6453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "7b543de346274816a1547de7a70242c0"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "0a0169f359eff97c8293b40644401d8f"
server_key_iv: "1386c0e0a236bae6335eb1d3fae77a99"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.527805 6453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:09.549932 6453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:09.553239 6453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:09.553442 6453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:09.554026 6453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:09.554817 6453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:09.554886 6453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:09.554952 6453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:09.555003 6453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:09.564015 6453 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:33459
I20260925 22:46:09.564179 6581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:33459 every 8 connection(s)
I20260925 22:46:09.565011 6453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:09.568753 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 6453
I20260925 22:46:09.568886 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:09.569190 806 external_mini_cluster.cc:1428] Setting key 202b43d973c5d356a8b99e2c6e6a37a5
Sep 25 22:46:09 dist-test-slave-vdtf krb5kdc[6346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:46:09.579447 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:09.566904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:53519 (local address 127.0.201.190:42519)
0925 22:46:09.567044 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:09.567053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:09.567776 (+ 723us) server_negotiation.cc:407] Connection header received
0925 22:46:09.569499 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:09.569501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:09.569549 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:09.569646 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:09.571073 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:09.571579 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:09.573026 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:09.573279 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:09.573400 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:09.575887 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:09.575891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:09.575894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:09.575920 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:09.577376 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:09.577916 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:09.577921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:09.577924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:09.578026 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:09.578393 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:09.578395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:09.578396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:09.578578 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:09.578652 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:09.579228 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 22:46:09.579327 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:46:09.580509 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:09.567176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42519 (local address 127.0.201.129:53519)
0925 22:46:09.567595 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:09.567638 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:09.569242 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:09.569814 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:09.569821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:09.570283 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:09.570916 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:09.570944 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:09.571736 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:09.571743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:09.572900 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:09.572909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:09.573432 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:09.573436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:09.573601 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:09.574238 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:09.574263 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:09.575720 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:09.577502 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:09.577507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:09.577521 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:09.577768 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:09.578152 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:09.578155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:09.578156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:09.578291 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:09.578697 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:09.578732 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:09.579012 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 22:46:09.579165 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:46:09.579406 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260925 22:46:09.581858 6582 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42519
I20260925 22:46:09.582150 6582 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:09.582720 6582 heartbeater.cc:511] Master 127.0.201.190:42519 requested a full tablet report, sending...
I20260925 22:46:09.584311 6391 ts_manager.cc:194] Registered new tserver with Master: 7b543de346274816a1547de7a70242c0 (127.0.201.129:33459)
I20260925 22:46:09.585431 6391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:53519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:46:09.595551 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42519
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:09.715366 6587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:09.715644 6587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:09.715745 6587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:09.719823 6587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:09.719893 6587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:09.719990 6587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:09.725396 6587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:09.726393 6587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:09.727375 6587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:09.736533 6593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:09.736521 6592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:09.736521 6595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:09.736789 6587 server_base.cc:1034] running on GCE node
I20260925 22:46:09.737187 6587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:09.737800 6587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:09.738952 6587 hybrid_clock.cc:648] HybridClock initialized: now 1790376369738923 us; error 42 us; skew 500 ppm
Sep 25 22:46:09 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:09.741618 6587 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:46:09.742452 6587 webserver.cc:460] Webserver started at http://127.0.201.130:44383/ using document root <none> and password file <none>
I20260925 22:46:09.742981 6587 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:09.743037 6587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:09.743283 6587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:09.745234 6587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "225e9976e86d433a8472f50a4b0d65d3"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "63be1ee1842953f81cf5034d25afe8b3"
server_key_iv: "79855f2ab5bbba2741e6c3e27f0f356b"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.745771 6587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "225e9976e86d433a8472f50a4b0d65d3"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "63be1ee1842953f81cf5034d25afe8b3"
server_key_iv: "79855f2ab5bbba2741e6c3e27f0f356b"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.749408 6587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 22:46:09.751797 6602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:09.752981 6587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 22:46:09.753113 6587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "225e9976e86d433a8472f50a4b0d65d3"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "63be1ee1842953f81cf5034d25afe8b3"
server_key_iv: "79855f2ab5bbba2741e6c3e27f0f356b"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.753213 6587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:09.771811 6587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:09.775008 6587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:09.775207 6587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:09.775861 6587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:09.776676 6587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:09.776721 6587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:09.776759 6587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:09.776774 6587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:09.785455 6587 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:37505
I20260925 22:46:09.785480 6715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:37505 every 8 connection(s)
I20260925 22:46:09.786291 6587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:09.791642 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 6587
I20260925 22:46:09.791800 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:09.792107 806 external_mini_cluster.cc:1428] Setting key 499434cbae0379d236df29670f85c299
Sep 25 22:46:09 dist-test-slave-vdtf krb5kdc[6346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:09.800215 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:09.788073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:35507 (local address 127.0.201.190:42519)
0925 22:46:09.788199 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:09.788202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:09.788993 (+ 791us) server_negotiation.cc:407] Connection header received
0925 22:46:09.790374 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:09.790377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:09.790432 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:09.790529 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:09.791629 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:09.792378 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:09.793672 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:09.793899 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:09.794006 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:09.796572 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:09.796575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:09.796577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:09.796599 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:09.797988 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:09.798612 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:09.798616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:09.798617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:09.798695 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:09.799111 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:09.799113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:09.799115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:09.799278 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:09.799352 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:09.800006 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 22:46:09.800100 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:46:09.801373 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:09.788453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42519 (local address 127.0.201.130:35507)
0925 22:46:09.788856 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:09.788896 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:09.790160 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:09.790696 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:09.790702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:09.791038 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:09.791489 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:09.791502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:09.792499 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:09.792507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:09.793537 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:09.793546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:09.794038 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:09.794042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:09.794202 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:09.794775 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:09.794793 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:09.796412 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:09.798115 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:09.798123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:09.798140 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:09.798506 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:09.798806 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:09.798812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:09.798815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:09.799008 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:09.799380 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:09.799389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:09.799745 (+ 356us) client_negotiation.cc:769] Sending connection context
0925 22:46:09.799979 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 22:46:09.800326 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:46:09.802542 6716 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42519
I20260925 22:46:09.802765 6716 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:09.803301 6716 heartbeater.cc:511] Master 127.0.201.190:42519 requested a full tablet report, sending...
I20260925 22:46:09.804390 6391 ts_manager.cc:194] Registered new tserver with Master: 225e9976e86d433a8472f50a4b0d65d3 (127.0.201.130:37505)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
I20260925 22:46:09.805063 6391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:35507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:46:09.819229 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42519
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:09.932291 6721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:09.932574 6721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:09.932664 6721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:09.936687 6721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:09.936761 6721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:09.936852 6721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:09.941999 6721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:09.943061 6721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:09.944170 6721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:09.953505 6726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:09.953500 6729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:09.953557 6727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:09.953780 6721 server_base.cc:1034] running on GCE node
I20260925 22:46:09.954241 6721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:09.954862 6721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:09.956014 6721 hybrid_clock.cc:648] HybridClock initialized: now 1790376369955983 us; error 43 us; skew 500 ppm
Sep 25 22:46:09 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:09.958724 6721 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:46:09.959558 6721 webserver.cc:460] Webserver started at http://127.0.201.131:44061/ using document root <none> and password file <none>
I20260925 22:46:09.960119 6721 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:09.960177 6721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:09.960391 6721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:09.962278 6721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "557f537867f645c3b45081d60df46f07"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "50cb9456cfd4da8b77e305b06c064b6d"
server_key_iv: "d7071f2403103f41206e17cfdff4fb5b"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.962774 6721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "557f537867f645c3b45081d60df46f07"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "50cb9456cfd4da8b77e305b06c064b6d"
server_key_iv: "d7071f2403103f41206e17cfdff4fb5b"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.966409 6721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:46:09.968704 6736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:09.969870 6721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:09.969959 6721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "557f537867f645c3b45081d60df46f07"
format_stamp: "Formatted at 2026-09-25 22:46:09 on dist-test-slave-vdtf"
server_key: "50cb9456cfd4da8b77e305b06c064b6d"
server_key_iv: "d7071f2403103f41206e17cfdff4fb5b"
server_key_version: "encryptionkey@0"
I20260925 22:46:09.970034 6721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:09.993263 6721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:09.996191 6721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:09.996353 6721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:09.996886 6721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:09.997663 6721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:09.997716 6721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:09.997761 6721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:09.997776 6721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:10.006142 6721 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:38255
I20260925 22:46:10.006157 6849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:38255 every 8 connection(s)
I20260925 22:46:10.007006 6721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:10.015573 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 6721
I20260925 22:46:10.015707 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:46:10.016053 806 external_mini_cluster.cc:1428] Setting key 7ae1be7ce5fef0a15dc92f9a462c6147
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:10.020666 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:10.008808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:40717 (local address 127.0.201.190:42519)
0925 22:46:10.008942 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:10.008945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:10.009783 (+ 838us) server_negotiation.cc:407] Connection header received
0925 22:46:10.011119 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:10.011121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:10.011168 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:10.011245 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:10.012497 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:10.013028 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:10.014120 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:10.014353 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:10.014429 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:10.017207 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:10.017211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:10.017214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:10.017241 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:10.018792 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:10.019232 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:10.019234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:10.019235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:10.019301 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:10.019618 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:10.019620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:10.019621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:10.019874 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:10.019976 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:10.020462 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 22:46:10.020553 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:46:10.021581 6852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:10.009185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42519 (local address 127.0.201.131:40717)
0925 22:46:10.009636 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:10.009677 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:10.010912 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:10.011458 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:10.011464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:10.011890 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:10.012351 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:10.012373 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:10.013142 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:10.013144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:10.014017 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:10.014026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:10.014482 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:10.014485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:10.014623 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:10.015225 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:10.015241 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:10.017037 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:10.018895 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:10.018900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:10.018911 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:10.019151 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:10.019417 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:10.019420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:10.019421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:10.019542 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:10.019981 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:10.019990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:10.020228 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 22:46:10.020417 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 22:46:10.020660 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":103,"threads_started":1}
I20260925 22:46:10.022717 6850 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42519
I20260925 22:46:10.022969 6850 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:10.023458 6850 heartbeater.cc:511] Master 127.0.201.190:42519 requested a full tablet report, sending...
I20260925 22:46:10.024482 6391 ts_manager.cc:194] Registered new tserver with Master: 557f537867f645c3b45081d60df46f07 (127.0.201.131:38255)
I20260925 22:46:10.025046 6391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:40717
I20260925 22:46:10.029865 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/test-admin.keytab.
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:10.044204 806 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 22:46:10.046968 6857 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:10.045494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.0.201.129:33459)
0925 22:46:10.045852 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:10.045866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:10.045911 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:46:10.045959 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:10.045964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:10.046179 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:10.046298 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:10.046813 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59412: BlockingRecv error: recv got EOF from 127.0.0.1:59412 (error 108)
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
W20260925 22:46:10.047340 806 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:10.048359 806 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:10.049250 806 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:46:10.059057 6857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:10.050082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.0.201.129:33459)
0925 22:46:10.050212 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:10.050215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:10.050229 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:46:10.050338 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:10.050341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:10.050376 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:10.050442 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:10.050939 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:10.051779 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:10.052621 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:10.053000 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:10.053083 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:10.055308 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:10.055318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:10.055345 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:10.055369 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:10.057378 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:10.057785 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:10.057791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:10.057792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:10.057875 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:10.058141 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:10.058143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:10.058144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:10.058369 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:10.058516 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:10.058715 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:46:10.058901 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:46:10.062963 6857 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:10.061918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.0.201.129:33459)
0925 22:46:10.062048 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:10.062053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:10.062161 (+ 108us) server_negotiation.cc:407] Connection header received
0925 22:46:10.062300 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:10.062303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:10.062341 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:10.062438 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:10.062833 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59442: BlockingRecv error: recv got EOF from 127.0.0.1:59442 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 22:46:10.063100 806 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:10.064128 806 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:10.065059 806 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:46:10.073127 6857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:10.065978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.0.201.129:33459)
0925 22:46:10.066090 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:10.066099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:10.066185 (+ 86us) server_negotiation.cc:407] Connection header received
0925 22:46:10.066261 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:10.066264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:10.066294 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:10.066360 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:10.066789 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:10.067298 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:10.068162 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:10.068377 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:10.068452 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:10.070438 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:10.070441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:10.070443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:10.070462 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:10.071622 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:10.072048 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:10.072051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:10.072052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:10.072141 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:10.072368 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:10.072370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:10.072372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:10.072528 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:10.072674 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:10.072846 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 22:46:10.072984 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 22:46:10.075762 6857 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:10.074764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.0.201.129:33459)
0925 22:46:10.074878 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:10.074882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:10.075065 (+ 183us) server_negotiation.cc:407] Connection header received
0925 22:46:10.075128 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:10.075130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:10.075169 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:10.075232 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:10.075627 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59464: BlockingRecv error: recv got EOF from 127.0.0.1:59464 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 22:46:10.076099 806 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376341615954-806-0/krb5kdc/krb5cc)
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:10.077068 806 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:10.077909 806 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:46:10.086324 6857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:10.078639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.0.201.129:33459)
0925 22:46:10.078756 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:10.078759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:10.078939 (+ 180us) server_negotiation.cc:407] Connection header received
0925 22:46:10.079012 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:10.079014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:10.079047 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:10.079111 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:10.079544 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:10.080073 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:10.080897 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:10.081113 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:10.081179 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:10.083467 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:10.083470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:10.083472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:10.083491 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:10.084802 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:10.085196 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:10.085199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:10.085200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:10.085274 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:10.085550 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:10.085553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:10.085554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:10.085714 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:10.085822 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:10.086023 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 22:46:10.086150 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:46:10.087337 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 6453
I20260925 22:46:10.092736 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 6587
I20260925 22:46:10.097767 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 6721
I20260925 22:46:10.102905 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 6359
2026-09-25T22:46:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2116 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6870](info): set up 2 sockets
Sep 25 22:46:10 dist-test-slave-vdtf krb5kdc[6870](info): commencing operation
krb5kdc: starting...
W20260925 22:46:11.132076 806 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:46:11 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.190.keytab.
Entry for principal oryx/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.190.keytab.
I20260925 22:46:11.250214 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:33667
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.190.keytab
--principal=oryx/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:33667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:11.357048 6883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:11.357292 6883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:11.357332 6883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:11.361279 6883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:11.361334 6883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:11.361354 6883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:11.361371 6883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:11.361392 6883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:11.366441 6883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:33667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.190.keytab
--principal=oryx/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:33667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:11.367377 6883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:11.368398 6883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:11.376689 6889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:11.376710 6888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:11.376724 6891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:11.376893 6883 server_base.cc:1034] running on GCE node
I20260925 22:46:11.377602 6883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:11.378741 6883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:11.379931 6883 hybrid_clock.cc:648] HybridClock initialized: now 1790376371379918 us; error 37 us; skew 500 ppm
Sep 25 22:46:11 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:11.387817 6883 init.cc:377] Logged in from keytab as oryx/127.0.201.190@KRBTEST.COM (short username oryx)
I20260925 22:46:11.388700 6883 webserver.cc:460] Webserver started at http://127.0.201.190:40041/ using document root <none> and password file <none>
I20260925 22:46:11.389262 6883 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:11.389393 6883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:11.389622 6883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:11.391901 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "16a9d45479b44fefbc4803169474a9a5"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "0184ae7e29b06a814f373e02687ddf81"
server_key_iv: "326e517c9dcdb91b248a8f19aeb8ec59"
server_key_version: "encryptionkey@0"
I20260925 22:46:11.392551 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "16a9d45479b44fefbc4803169474a9a5"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "0184ae7e29b06a814f373e02687ddf81"
server_key_iv: "326e517c9dcdb91b248a8f19aeb8ec59"
server_key_version: "encryptionkey@0"
I20260925 22:46:11.396178 6883 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:46:11.398523 6898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:11.399885 6883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:11.400080 6883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "16a9d45479b44fefbc4803169474a9a5"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "0184ae7e29b06a814f373e02687ddf81"
server_key_iv: "326e517c9dcdb91b248a8f19aeb8ec59"
server_key_version: "encryptionkey@0"
I20260925 22:46:11.400205 6883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:11.426653 6883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:11.429561 6883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:11.429766 6883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:11.436694 6883 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:33667
I20260925 22:46:11.437001 6950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:33667 every 8 connection(s)
I20260925 22:46:11.437654 6883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:11.440421 6951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:11.445784 6951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: Bootstrap starting.
I20260925 22:46:11.446193 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 6883
I20260925 22:46:11.446377 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:11.446683 806 external_mini_cluster.cc:1428] Setting key 2bae8454039a40ab651d14284257f5ab
I20260925 22:46:11.448483 6951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:11.449297 6951 log.cc:824] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:11.451455 6951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: No bootstrap required, opened a new log
I20260925 22:46:11.455228 6951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } }
I20260925 22:46:11.455456 6951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:11.455533 6951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a9d45479b44fefbc4803169474a9a5, State: Initialized, Role: FOLLOWER
Sep 25 22:46:11 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:11.456018 6951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } }
I20260925 22:46:11.456164 6951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:11.456245 6951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:11.456339 6951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:11.457523 6951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } }
I20260925 22:46:11.457888 6951 leader_election.cc:302] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16a9d45479b44fefbc4803169474a9a5; no voters:
I20260925 22:46:11.458205 6951 leader_election.cc:288] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:11.458343 6956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:11.458606 6956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 1 LEADER]: Becoming Leader. State: Replica: 16a9d45479b44fefbc4803169474a9a5, State: Running, Role: LEADER
I20260925 22:46:11.458949 6956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } }
I20260925 22:46:11.459522 6951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:11.460569 6958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a9d45479b44fefbc4803169474a9a5. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } }
I20260925 22:46:11.460687 6958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:11.461004 6957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } }
I20260925 22:46:11.461077 6957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:11.462452 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:11.447998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.0.201.190:33667)
0925 22:46:11.448398 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:11.448413 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:11.448447 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:46:11.449239 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:11.449259 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:11.449634 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:11.449907 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:11.450664 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:11.452104 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:11.452956 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:11.453643 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:11.453799 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:11.456556 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:11.456566 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:11.456581 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:11.456613 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:11.458518 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:11.459002 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:11.459007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:11.459013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:11.459135 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:11.459448 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:11.459451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:11.459453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:11.459961 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:11.460866 (+ 905us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:11.461012 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 22:46:11.461339 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":128,"threads_started":1}
W20260925 22:46:11.464440 6971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:46:11.464516 6971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:46:11.464569 6965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:11.465381 6965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:11.470741 6965 catalog_manager.cc:1347] Generated new cluster ID: df6895d752294032b572668a61c7b7c7
I20260925 22:46:11.470808 6965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:11.492681 6965 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:11.493654 6965 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:11.503886 6965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: Generated new TSK 0
I20260925 22:46:11.504673 6965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab.
Entry for principal oryx/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab.
I20260925 22:46:11.536706 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33667
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab
--principal=oryx/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:11.651216 6977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:11.651511 6977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:11.651589 6977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:11.655515 6977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:11.655608 6977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:11.655715 6977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:11.660977 6977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab
--principal=oryx/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:11.662029 6977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:11.662997 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:11.672124 6982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:11.672108 6983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:11.672206 6977 server_base.cc:1034] running on GCE node
W20260925 22:46:11.672130 6985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:11.672721 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:11.673346 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:11.674525 6977 hybrid_clock.cc:648] HybridClock initialized: now 1790376371674496 us; error 46 us; skew 500 ppm
Sep 25 22:46:11 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:11.677210 6977 init.cc:377] Logged in from keytab as oryx/127.0.201.129@KRBTEST.COM (short username oryx)
I20260925 22:46:11.677934 6977 webserver.cc:460] Webserver started at http://127.0.201.129:43505/ using document root <none> and password file <none>
I20260925 22:46:11.678411 6977 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:11.678471 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:11.678629 6977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:11.680552 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "427fbaeacb8b4c66ad3417551bc7174c"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "19e29733aaafd5ff8d45a47d425fe68e"
server_key_iv: "bf6685dfbf6f6101a118a2b7da761710"
server_key_version: "encryptionkey@0"
I20260925 22:46:11.681075 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "427fbaeacb8b4c66ad3417551bc7174c"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "19e29733aaafd5ff8d45a47d425fe68e"
server_key_iv: "bf6685dfbf6f6101a118a2b7da761710"
server_key_version: "encryptionkey@0"
I20260925 22:46:11.684638 6977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:46:11.686815 6992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:11.688020 6977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:11.688114 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "427fbaeacb8b4c66ad3417551bc7174c"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "19e29733aaafd5ff8d45a47d425fe68e"
server_key_iv: "bf6685dfbf6f6101a118a2b7da761710"
server_key_version: "encryptionkey@0"
I20260925 22:46:11.688192 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:11.725031 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:11.728477 6977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:11.728633 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:11.729166 6977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:11.729931 6977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:11.729977 6977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:11.730016 6977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:11.730032 6977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:11.738487 6977 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:44531
I20260925 22:46:11.738519 7105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:44531 every 8 connection(s)
I20260925 22:46:11.739346 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:11.743420 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 6977
I20260925 22:46:11.743603 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:11.743894 806 external_mini_cluster.cc:1428] Setting key 33c8bd198085ffd5a76f8e576875cca4
Sep 25 22:46:11 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:11.753719 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:11.741124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:33753 (local address 127.0.201.190:33667)
0925 22:46:11.741268 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:11.741278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:11.742072 (+ 794us) server_negotiation.cc:407] Connection header received
0925 22:46:11.743441 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:11.743444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:11.743500 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:11.743581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:11.744912 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:11.745557 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:11.746681 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:11.746929 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:11.747046 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:11.749900 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:11.749906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:11.749913 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:11.749944 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:11.751639 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:11.752148 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:11.752153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:11.752155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:11.752254 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:11.752600 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:11.752603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:11.752605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:11.752837 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:11.752957 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:11.753424 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 22:46:11.753554 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for oryx/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:46:11.754395 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:11.741485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.129:33753)
0925 22:46:11.741919 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:11.741962 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:11.743220 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:11.743779 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:11.743787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:11.744195 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:11.744764 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:11.744778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:11.745680 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:11.745683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:11.746551 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:11.746560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:11.747080 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:11.747084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:11.747252 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:11.747862 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:11.747879 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:11.749696 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:11.751776 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:11.751781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:11.751793 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:11.752043 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:11.752355 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:11.752357 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:11.752359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:11.752504 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:11.752946 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:11.752951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:11.753166 (+ 215us) client_negotiation.cc:769] Sending connection context
0925 22:46:11.753319 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:46:11.753584 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260925 22:46:11.755625 7106 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33667
I20260925 22:46:11.755867 7106 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:11.756301 7106 heartbeater.cc:511] Master 127.0.201.190:33667 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.130@KRBTEST.COM" created.
I20260925 22:46:11.757812 6915 ts_manager.cc:194] Registered new tserver with Master: 427fbaeacb8b4c66ad3417551bc7174c (127.0.201.129:44531)
I20260925 22:46:11.758885 6915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:33753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab.
Entry for principal oryx/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab.
I20260925 22:46:11.772709 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33667
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab
--principal=oryx/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:11.883085 7111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:11.883379 7111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:11.883463 7111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:11.887416 7111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:11.887511 7111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:11.887599 7111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:11.892941 7111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab
--principal=oryx/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:11.893973 7111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:11.894979 7111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:11.904538 7116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:11.904577 7117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:11.904551 7111 server_base.cc:1034] running on GCE node
W20260925 22:46:11.904556 7119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:11.905288 7111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:11.905838 7111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:11.907033 7111 hybrid_clock.cc:648] HybridClock initialized: now 1790376371907019 us; error 24 us; skew 500 ppm
Sep 25 22:46:11 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:11.909801 7111 init.cc:377] Logged in from keytab as oryx/127.0.201.130@KRBTEST.COM (short username oryx)
I20260925 22:46:11.910631 7111 webserver.cc:460] Webserver started at http://127.0.201.130:33731/ using document root <none> and password file <none>
I20260925 22:46:11.911121 7111 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:11.911177 7111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:11.911331 7111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:11.913303 7111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "7cd22e79ff714a928cd1116f0261e49e"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "3cfbf779d6fe265fe6a995cce488d2dc"
server_key_iv: "b86a7e9d10861e055b5c4a14638649e2"
server_key_version: "encryptionkey@0"
I20260925 22:46:11.913815 7111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "7cd22e79ff714a928cd1116f0261e49e"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "3cfbf779d6fe265fe6a995cce488d2dc"
server_key_iv: "b86a7e9d10861e055b5c4a14638649e2"
server_key_version: "encryptionkey@0"
I20260925 22:46:11.917296 7111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:46:11.919643 7126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:11.920938 7111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:46:11.921029 7111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "7cd22e79ff714a928cd1116f0261e49e"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "3cfbf779d6fe265fe6a995cce488d2dc"
server_key_iv: "b86a7e9d10861e055b5c4a14638649e2"
server_key_version: "encryptionkey@0"
I20260925 22:46:11.921157 7111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:11.939385 7111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:11.942507 7111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:11.942709 7111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:11.943310 7111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:11.944191 7111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:11.944262 7111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:11.944331 7111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:11.944392 7111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:11.953402 7111 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:32939
I20260925 22:46:11.953433 7239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:32939 every 8 connection(s)
I20260925 22:46:11.954285 7111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:11.958451 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 7111
I20260925 22:46:11.958585 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:11.958894 806 external_mini_cluster.cc:1428] Setting key 16d1dd53fcd40c75cc83bfe6cea2f8f6
Sep 25 22:46:11 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:46:11.968559 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:11.956085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:40131 (local address 127.0.201.190:33667)
0925 22:46:11.956244 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:11.956248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:11.957072 (+ 824us) server_negotiation.cc:407] Connection header received
0925 22:46:11.958356 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:11.958361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:11.958419 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:11.958521 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:11.959947 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:11.960645 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:11.962254 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:11.962486 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:11.962596 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:11.965041 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:11.965044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:11.965047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:11.965077 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:11.966504 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:11.967009 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:11.967013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:11.967014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:11.967094 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:11.967427 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:11.967432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:11.967434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:11.967615 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:11.967748 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:11.968275 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 22:46:11.968390 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:46:11.969394 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:11.956492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.130:40131)
0925 22:46:11.956921 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:11.956961 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:11.958142 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:11.958731 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:11.958738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:11.959141 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:11.959774 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:11.959790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:11.960801 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:11.960805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:11.962136 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:11.962145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:11.962627 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:11.962631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:11.962792 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:11.963364 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:11.963409 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:11.964886 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:11.966633 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:11.966639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:11.966653 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:11.966901 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:11.967192 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:11.967195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:11.967197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:11.967326 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:11.967758 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:11.967766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:11.968030 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 22:46:11.968223 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 22:46:11.968535 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.131@KRBTEST.COM" created.
I20260925 22:46:11.971762 7240 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33667
I20260925 22:46:11.972028 7240 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:11.972623 7240 heartbeater.cc:511] Master 127.0.201.190:33667 requested a full tablet report, sending...
I20260925 22:46:11.973739 6915 ts_manager.cc:194] Registered new tserver with Master: 7cd22e79ff714a928cd1116f0261e49e (127.0.201.130:32939)
I20260925 22:46:11.974288 6915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:40131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab.
Entry for principal oryx/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab.
I20260925 22:46:11.986075 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33667
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab
--principal=oryx/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:12.095811 7245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:12.096076 7245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:12.096118 7245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:12.100091 7245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:12.100148 7245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:12.100206 7245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:12.105795 7245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab
--principal=oryx/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:12.106768 7245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:12.107774 7245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:12.117399 7253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.117395 7251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.117393 7250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:12.117676 7245 server_base.cc:1034] running on GCE node
I20260925 22:46:12.118180 7245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:12.118829 7245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:12.120036 7245 hybrid_clock.cc:648] HybridClock initialized: now 1790376372120028 us; error 31 us; skew 500 ppm
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:12.123059 7245 init.cc:377] Logged in from keytab as oryx/127.0.201.131@KRBTEST.COM (short username oryx)
I20260925 22:46:12.123991 7245 webserver.cc:460] Webserver started at http://127.0.201.131:37063/ using document root <none> and password file <none>
I20260925 22:46:12.124579 7245 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:12.124667 7245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:12.124922 7245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:12.126967 7245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "f4af0e5f397c45908604223c8fff2b4f"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "cc1a9c7fb3bcd2f1c9ee7ef40e70b1f5"
server_key_iv: "efa1dac04bc3e892dadee5dfc9c2c7ad"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.127615 7245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "f4af0e5f397c45908604223c8fff2b4f"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "cc1a9c7fb3bcd2f1c9ee7ef40e70b1f5"
server_key_iv: "efa1dac04bc3e892dadee5dfc9c2c7ad"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.131292 7245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:46:12.133795 7260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.135046 7245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:12.135185 7245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "f4af0e5f397c45908604223c8fff2b4f"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "cc1a9c7fb3bcd2f1c9ee7ef40e70b1f5"
server_key_iv: "efa1dac04bc3e892dadee5dfc9c2c7ad"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.135300 7245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:12.154053 7245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:12.157362 7245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:12.157567 7245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:12.158150 7245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:12.158957 7245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:12.159027 7245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.159096 7245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:12.159132 7245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.168130 7245 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:36793
I20260925 22:46:12.168154 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:36793 every 8 connection(s)
I20260925 22:46:12.169050 7245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:12.172466 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 7245
I20260925 22:46:12.172561 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:46:12.172864 806 external_mini_cluster.cc:1428] Setting key e630b6559996f8dbe3c454de245a9bdf
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:12.182065 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.170771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:60713 (local address 127.0.201.190:33667)
0925 22:46:12.170902 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.170905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.171700 (+ 795us) server_negotiation.cc:407] Connection header received
0925 22:46:12.173179 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.173182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.173235 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.173328 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.174653 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.175296 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.176476 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.176688 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.176764 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.179080 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.179083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.179086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.179108 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.180415 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.180843 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.180845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.180846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.180915 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.181193 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.181195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.181196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.181345 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.181415 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.181835 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.181926 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:46:12.182890 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.171116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.131:60713)
0925 22:46:12.171537 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.171575 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.172968 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.173515 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.173521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.173974 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.174524 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.174537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.175402 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.175439 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.176362 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.176371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.176807 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.176809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.176943 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.177537 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.177552 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.178938 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.180510 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.180515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.180526 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.180760 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.180991 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.180993 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.180995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.181116 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.181431 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.181436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.181643 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.181790 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.182022 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":112,"threads_started":1}
I20260925 22:46:12.184026 7374 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33667
I20260925 22:46:12.184252 7374 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:12.184778 7374 heartbeater.cc:511] Master 127.0.201.190:33667 requested a full tablet report, sending...
I20260925 22:46:12.185868 6914 ts_manager.cc:194] Registered new tserver with Master: f4af0e5f397c45908604223c8fff2b4f (127.0.201.131:36793)
I20260925 22:46:12.186650 6914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:60713
I20260925 22:46:12.186812 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.201.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.189.keytab.
Entry for principal oryx/127.0.201.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.189.keytab.
I20260925 22:46:12.214214 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.189:35691
--webserver_interface=127.0.201.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.189.keytab
--principal=oryx/127.0.201.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:33667,127.0.201.189:35691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:12.323478 7379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:12.323751 7379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:12.323810 7379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:12.327917 7379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:12.327968 7379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:12.327987 7379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:12.328008 7379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:12.328024 7379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:12.333089 7379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:33667,127.0.201.189:35691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.189.keytab
--principal=oryx/127.0.201.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.189:35691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.201.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
W20260925 22:46:12.333253 7379 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.201.190:33667,127.0.201.189:35691'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 22:46:12.334017 7379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:12.334910 7379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:12.343251 7387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.343258 7385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.343267 7384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:12.343291 7379 server_base.cc:1034] running on GCE node
I20260925 22:46:12.344062 7379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:12.345021 7379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:12.346175 7379 hybrid_clock.cc:648] HybridClock initialized: now 1790376372346157 us; error 30 us; skew 500 ppm
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:12.348730 7379 init.cc:377] Logged in from keytab as oryx/127.0.201.189@KRBTEST.COM (short username oryx)
I20260925 22:46:12.349468 7379 webserver.cc:460] Webserver started at http://127.0.201.189:43357/ using document root <none> and password file <none>
I20260925 22:46:12.349948 7379 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:12.350001 7379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:12.350147 7379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:12.352073 7379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data/instance:
uuid: "6daee056e6dd48e2a9d24fd3451b9460"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "3c713518e34f97616e5a17687f1b0d3c"
server_key_iv: "c8d1e4098d5db8f9e47ccc971f428cca"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.352562 7379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal/instance:
uuid: "6daee056e6dd48e2a9d24fd3451b9460"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "3c713518e34f97616e5a17687f1b0d3c"
server_key_iv: "c8d1e4098d5db8f9e47ccc971f428cca"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.355938 7379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:46:12.358071 7394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.359217 7379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:12.359308 7379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal
uuid: "6daee056e6dd48e2a9d24fd3451b9460"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "3c713518e34f97616e5a17687f1b0d3c"
server_key_iv: "c8d1e4098d5db8f9e47ccc971f428cca"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.359452 7379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:12.381294 7379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:12.384156 7379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:12.384343 7379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:12.390923 7379 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.189:35691
I20260925 22:46:12.391219 7446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.189:35691 every 8 connection(s)
I20260925 22:46:12.391987 7379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
I20260925 22:46:12.394614 7447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:12.398144 7447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:46:12.400259 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 7379
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.189@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.201.189@KRBTEST.COM
I20260925 22:46:12.412206 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.399927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.0.201.190:33667)
0925 22:46:12.400084 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.400087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.401025 (+ 938us) server_negotiation.cc:407] Connection header received
0925 22:46:12.402038 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.402042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.402098 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.402197 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.402962 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.403514 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.404857 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.405079 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.405155 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.407801 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.407810 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.407812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.407834 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.409777 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.410405 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.410409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.410414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.410511 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.410905 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.410908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.410909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.411166 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.411275 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.411886 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.412012 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:46:12.413096 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.400346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:54306)
0925 22:46:12.400873 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.400929 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.401822 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.402244 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.402250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.402593 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.402819 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.402831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.403654 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.403659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.404703 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.404713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.405239 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.405243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.405388 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.406021 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.406043 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.407646 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.409906 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.409914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.409921 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.410286 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.410603 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.410607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.410609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.410789 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.411304 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.411308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.411565 (+ 257us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.412392 (+ 827us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.412551 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":28,"spinlock_wait_cycles":896,"thread_start_us":147,"threads_started":1}
I20260925 22:46:12.413097 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.401120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.0.201.189:35691)
0925 22:46:12.401492 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.401501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.401540 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:46:12.401837 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.401856 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.402046 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.402228 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.403078 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.404137 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.405226 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.405590 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.405705 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.408517 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.408525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.408540 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.408568 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.410363 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.410898 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.410903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.410905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.410985 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.411248 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.411252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.411253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.411550 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.411764 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.411998 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.412253 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260925 22:46:12.414492 7447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.189@KRBTEST.COM for oryx/127.0.201.189@KRBTEST.COM
I20260925 22:46:12.422508 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.415021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:35691 (local address 127.0.0.1:39354)
0925 22:46:12.415140 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.415155 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.415224 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.415524 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.415526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.415654 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.415806 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.415811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.416561 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.416564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.417131 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.417139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.417607 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.417610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.417697 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.418192 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.418204 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.419482 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.421367 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.421369 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.421370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.421557 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.421789 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.421791 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.421792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.421860 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.422166 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.422168 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.422236 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.422312 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.422400 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 22:46:12.422595 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.415229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.0.201.189:35691)
0925 22:46:12.415350 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.415353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.415363 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:46:12.415390 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.415392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.415426 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.415492 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.415936 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.416440 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.417251 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.417489 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.417588 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.419604 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.419607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.419609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.419630 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.421261 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.421641 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.421643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.421644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.421712 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.421934 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.421937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.421938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.422090 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.422166 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.422379 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.422464 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:46:12.426086 7447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: Bootstrap starting.
I20260925 22:46:12.428503 7447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:12.429246 7447 log.cc:824] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:12.431108 7447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: No bootstrap required, opened a new log
I20260925 22:46:12.434044 7447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } }
I20260925 22:46:12.434229 7447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:12.434290 7447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6daee056e6dd48e2a9d24fd3451b9460, State: Initialized, Role: FOLLOWER
I20260925 22:46:12.434688 7447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } }
I20260925 22:46:12.435132 7453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } } }
I20260925 22:46:12.435256 7453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:46:12.435914 7447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:46:12.440620 7464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:46:12.440739 7464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:46:12.442248 7379 master_runner.cc:428] Detected that this master 6daee056e6dd48e2a9d24fd3451b9460 is joining an existing cluster
I20260925 22:46:12.442302 7379 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.201.189:35691
I20260925 22:46:12.453465 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.444430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:35691 (local address 127.0.0.1:39356)
0925 22:46:12.445383 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.445404 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.445510 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.445785 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.445788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.446003 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.446109 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.446117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.447073 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.447075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.448097 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.448106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.448726 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.448729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.448806 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.449274 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.449288 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.449763 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.451937 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.451942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.451944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.452173 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.452456 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.452459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.452461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.452534 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.452946 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.452949 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.453028 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.453130 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.453252 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":57,"thread_start_us":85,"threads_started":1}
I20260925 22:46:12.453483 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.444732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39356 (local address 127.0.201.189:35691)
0925 22:46:12.444890 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.444895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.445464 (+ 569us) server_negotiation.cc:407] Connection header received
0925 22:46:12.445593 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.445596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.445631 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.445676 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.446220 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.446971 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.448206 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.448534 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.448596 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.449881 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.449884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.449887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.449906 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.451814 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.452274 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.452277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.452279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.452351 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.452617 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.452620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.452621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.452813 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.452882 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.453194 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.453312 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:46:12.453867 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.444834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.0.201.190:33667)
0925 22:46:12.444936 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.444940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.445180 (+ 240us) server_negotiation.cc:407] Connection header received
0925 22:46:12.445293 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.445296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.445346 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.445437 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.446086 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.446915 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.448224 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.448571 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.448648 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.450205 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.450208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.450211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.450234 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.452047 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.452538 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.452541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.452543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.452624 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.452958 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.452962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.452964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.453182 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.453278 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.453586 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.453689 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:46:12.454360 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.444392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:54314)
0925 22:46:12.445088 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.445101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.445201 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.445566 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.445569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.445805 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.445954 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.445962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.447052 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.447055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.448091 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.448101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.448793 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.448797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.448891 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.449458 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.449473 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.450040 (+ 567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.452201 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.452204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.452206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.452433 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.452737 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.452740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.452742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.452826 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.453319 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.453321 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.453400 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.454065 (+ 665us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.454194 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":37,"spinlock_wait_cycles":1792,"thread_start_us":88,"threads_started":1}
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.190@KRBTEST.COM for oryx/127.0.201.189@KRBTEST.COM
I20260925 22:46:12.468369 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.458689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:35691 (local address 127.0.0.1:39362)
0925 22:46:12.459022 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.459067 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.459186 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.459507 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.459511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.459881 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.459984 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.459989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.461010 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.461013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.461708 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.461717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.462260 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.462262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.462392 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.462853 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.462864 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.464471 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.466561 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.466568 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.466575 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.466896 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.467191 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.467197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.467202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.467303 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.467723 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.467733 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.467872 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.468037 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.468244 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260925 22:46:12.468832 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.458719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.0.201.189:35691)
0925 22:46:12.458818 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.458823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.459125 (+ 302us) server_negotiation.cc:407] Connection header received
0925 22:46:12.459333 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.459336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.459394 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.459473 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.460085 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.460914 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.461810 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.462154 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.462243 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.464597 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.464601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.464604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.464627 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.466454 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.467008 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.467012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.467014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.467093 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.467403 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.467406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.467408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.467592 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.467718 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.468583 (+ 865us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.468684 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:46:12.470182 6914 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.201.189:35691: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460"
member_type: NON_VOTER
last_known_addr {
host: "127.0.201.189"
port: 35691
}
attrs {
promote: true
}
}
dest_uuid: "16a9d45479b44fefbc4803169474a9a5"
cas_config_opid_index: -1
I20260925 22:46:12.470855 6914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } }
I20260925 22:46:12.472479 7475 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6daee056e6dd48e2a9d24fd3451b9460 (127.0.201.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } }
I20260925 22:46:12.473173 7421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:12.474151 7475 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.201.189:35691
I20260925 22:46:12.474267 7421 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Refusing update from remote peer 16a9d45479b44fefbc4803169474a9a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 22:46:12.474488 7474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } }
I20260925 22:46:12.474599 7474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:12.475028 7379 master.cc:537] Master@127.0.201.189:35691 shutting down...
I20260925 22:46:12.475011 7474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:46:12.476963 6902 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 22:46:12.477911 6902 consensus_peers.cc:487] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 -> Peer 6daee056e6dd48e2a9d24fd3451b9460 (127.0.201.189:35691): Couldn't send request to peer 6daee056e6dd48e2a9d24fd3451b9460. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 22:46:12.479125 7379 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:12.479386 7379 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:12.479483 7379 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: stopping tablet replica
WARNING: no policy specified for oryx/127.0.201.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.188@KRBTEST.COM" created.
I20260925 22:46:12.484263 7379 master.cc:552] Master@127.0.201.189:35691 shutdown complete.
I20260925 22:46:12.484342 7379 master_runner.cc:305] Clearing existing system tablet
I20260925 22:46:12.484593 7379 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:12.484668 7379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:12.485792 7379 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:46:12.486672 7480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.486909 7379 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:12.487068 7379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:12.487126 7379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal
uuid: "6daee056e6dd48e2a9d24fd3451b9460"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "3c713518e34f97616e5a17687f1b0d3c"
server_key_iv: "c8d1e4098d5db8f9e47ccc971f428cca"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.488929 7379 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:12.491046 7379 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:46:12.491149 7379 log.cc:1196] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 22:46:12.491427 7379 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: Deleting consensus metadata
I20260925 22:46:12.491618 7379 master_runner.cc:315] Copying system tablet from 127.0.201.190:33667
I20260925 22:46:12.492995 7379 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.201.190:33667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.188.keytab.
Entry for principal oryx/127.0.201.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.188.keytab.
I20260925 22:46:12.499547 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.188:45327
--webserver_interface=127.0.201.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.188.keytab
--principal=oryx/127.0.201.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:33667,127.0.201.189:35691,127.0.201.188:45327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:46:12.501633 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.493533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:54320)
0925 22:46:12.493813 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.493829 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.493946 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.494197 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.494200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.494419 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.494531 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.494538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.495403 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.495406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.496326 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.496339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.496857 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.496861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.496951 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.497458 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.497472 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.497873 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.499893 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.499898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.499902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.500147 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.500473 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.500478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.500481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.500568 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.501073 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.501077 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.501176 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.501287 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.501437 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
I20260925 22:46:12.502022 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.493652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.0.201.190:33667)
0925 22:46:12.493793 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.493798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.493897 (+ 99us) server_negotiation.cc:407] Connection header received
0925 22:46:12.494042 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.494044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.494088 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.494162 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.494650 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.495274 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.496457 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.496743 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.496837 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.498024 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.498028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.498030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.498052 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.499744 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.500259 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.500265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.500267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.500372 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.500675 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.500679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.500680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.500957 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.501072 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.501368 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.501507 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:46:12.502501 6935 tablet_copy_service.cc:140] P 16a9d45479b44fefbc4803169474a9a5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6daee056e6dd48e2a9d24fd3451b9460 ({username='oryx', principal='oryx/127.0.201.189@KRBTEST.COM'} at 127.0.0.1:54320)
I20260925 22:46:12.502684 6935 tablet_copy_service.cc:161] P 16a9d45479b44fefbc4803169474a9a5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6daee056e6dd48e2a9d24fd3451b9460 at {username='oryx', principal='oryx/127.0.201.189@KRBTEST.COM'} at 127.0.0.1:54320: session id = 6daee056e6dd48e2a9d24fd3451b9460-00000000000000000000000000000000
I20260925 22:46:12.504354 6935 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:46:12.505882 7379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:46:12.507777 7379 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: tablet copy: Starting download of 0 data blocks...
I20260925 22:46:12.508009 7379 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: tablet copy: Starting download of 1 WAL segments...
I20260925 22:46:12.509676 7379 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:46:12.511221 6935 tablet_copy_service.cc:342] P 16a9d45479b44fefbc4803169474a9a5: Request end of tablet copy session 6daee056e6dd48e2a9d24fd3451b9460-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.201.189@KRBTEST.COM'} at 127.0.0.1:54320
I20260925 22:46:12.511345 6935 tablet_copy_service.cc:433] P 16a9d45479b44fefbc4803169474a9a5: ending tablet copy session 6daee056e6dd48e2a9d24fd3451b9460-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6daee056e6dd48e2a9d24fd3451b9460
I20260925 22:46:12.518092 7379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:12.519760 7488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.519769 7491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:12.520341 7379 server_base.cc:1034] running on GCE node
W20260925 22:46:12.523768 7489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:12.523993 7379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:12.524329 7379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:12.525473 7379 hybrid_clock.cc:648] HybridClock initialized: now 1790376372525465 us; error 26 us; skew 500 ppm
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:12.526944 7379 init.cc:377] Logged in from keytab as oryx/127.0.201.189@KRBTEST.COM (short username oryx)
I20260925 22:46:12.527398 7379 webserver.cc:460] Webserver started at http://127.0.201.189:46035/ using document root <none> and password file <none>
I20260925 22:46:12.527585 7379 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:12.527640 7379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:12.528753 7379 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:46:12.529688 7498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.529912 7379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:46:12.530004 7379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal
uuid: "6daee056e6dd48e2a9d24fd3451b9460"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "3c713518e34f97616e5a17687f1b0d3c"
server_key_iv: "c8d1e4098d5db8f9e47ccc971f428cca"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.530088 7379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:12.535563 7379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:12.537496 7379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:12.542260 7379 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.189:35691
I20260925 22:46:12.542277 7550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.189:35691 every 8 connection(s)
I20260925 22:46:12.542865 7379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
I20260925 22:46:12.544819 7551 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:46:12.545583 7551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: Bootstrap starting.
I20260925 22:46:12.553143 7551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:46:12.553440 7551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: Bootstrap complete.
I20260925 22:46:12.553930 7551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } }
I20260925 22:46:12.554060 7551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6daee056e6dd48e2a9d24fd3451b9460, State: Initialized, Role: LEARNER
I20260925 22:46:12.554193 7551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } }
I20260925 22:46:12.554563 7554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } }
I20260925 22:46:12.554725 7551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:12.554725 7554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:46:12.557777 7565 catalog_manager.cc:1259] Loaded cluster ID: df6895d752294032b572668a61c7b7c7
I20260925 22:46:12.557888 7565 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: loading cluster ID for follower catalog manager: success
I20260925 22:46:12.559445 7565 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: acquiring CA information for follower catalog manager: success
I20260925 22:46:12.560752 7565 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:46:12.650198 7487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:12.650506 7487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:12.650586 7487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:12.654680 7487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:12.654762 7487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:12.654798 7487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:12.654834 7487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:12.654867 7487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:12.660560 7487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:33667,127.0.201.189:35691,127.0.201.188:45327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.188.keytab
--principal=oryx/127.0.201.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.188:45327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.201.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:12.661661 7487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:12.662657 7487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:12.670930 7573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.671128 7570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:12.671172 7487 server_base.cc:1034] running on GCE node
W20260925 22:46:12.670981 7571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:12.671816 7487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:12.673229 7487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:12.674453 7487 hybrid_clock.cc:648] HybridClock initialized: now 1790376372674437 us; error 32 us; skew 500 ppm
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:12.677280 7487 init.cc:377] Logged in from keytab as oryx/127.0.201.188@KRBTEST.COM (short username oryx)
I20260925 22:46:12.678131 7487 webserver.cc:460] Webserver started at http://127.0.201.188:44909/ using document root <none> and password file <none>
I20260925 22:46:12.678678 7487 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:12.678769 7487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:12.678997 7487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:12.681097 7487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data/instance:
uuid: "990efe788dd84cec88dad8e7cefbb68b"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "b73ea72a2fcfba4dc821d8b6fa1b9152"
server_key_iv: "d6663f036f5217b417f404fc2e008ba8"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.681645 7487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal/instance:
uuid: "990efe788dd84cec88dad8e7cefbb68b"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "b73ea72a2fcfba4dc821d8b6fa1b9152"
server_key_iv: "d6663f036f5217b417f404fc2e008ba8"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.685240 7487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:12.687465 7580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.688679 7487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:12.688818 7487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal
uuid: "990efe788dd84cec88dad8e7cefbb68b"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "b73ea72a2fcfba4dc821d8b6fa1b9152"
server_key_iv: "d6663f036f5217b417f404fc2e008ba8"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.688932 7487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:12.703784 7487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:12.706866 7487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:12.707065 7487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:12.713639 7487 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.188:45327
I20260925 22:46:12.713912 7632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.188:45327 every 8 connection(s)
I20260925 22:46:12.714588 7487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
I20260925 22:46:12.716806 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 7487
I20260925 22:46:12.717494 7633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:12.721506 7633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.201.188@KRBTEST.COM
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.188@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:12.732048 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.718295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.0.201.188:45327)
0925 22:46:12.718682 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.718696 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.718728 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:46:12.719352 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.719366 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.719790 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.720538 (+ 748us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.721339 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.722372 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.723233 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.723918 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.724092 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.726752 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.726760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.726774 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.726799 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.728703 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.729240 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.729246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.729251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.729379 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.729759 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.729763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.729765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.730319 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.730533 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.730742 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.731039 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":130,"threads_started":1}
I20260925 22:46:12.732681 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.723052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:54330)
0925 22:46:12.723457 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.723494 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.723626 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.723936 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.723942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.724319 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.724447 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.724453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.725297 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.725302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.726024 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.726033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.726567 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.726572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.726705 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.727170 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.727207 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.729118 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.731070 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.731077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.731081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.731283 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.731644 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.731646 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.731647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.731768 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.732189 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.732197 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.732312 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.732435 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.732565 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":144,"threads_started":1}
I20260925 22:46:12.732764 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.723208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.0.201.190:33667)
0925 22:46:12.723339 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.723344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.723569 (+ 225us) server_negotiation.cc:407] Connection header received
0925 22:46:12.723746 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.723749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.723799 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.723886 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.724563 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.725174 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.726145 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.726425 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.726536 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.729260 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.729263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.729266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.729292 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.730958 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.731461 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.731465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.731467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.731552 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.731857 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.731861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.731862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.732064 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.732166 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.732516 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.732625 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:46:12.734423 7633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.188@KRBTEST.COM for oryx/127.0.201.189@KRBTEST.COM
I20260925 22:46:12.742480 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.734857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:35691 (local address 127.0.0.1:39368)
0925 22:46:12.734970 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.734985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.735056 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.735407 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.735409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.735531 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.735597 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.735601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.736496 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.736499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.737014 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.737021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.737422 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.737425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.737483 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.737857 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.737876 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.739183 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.741230 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.741232 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.741234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.741415 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.741671 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.741673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.741674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.741741 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.742120 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.742122 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.742191 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.742267 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.742355 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 22:46:12.742615 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.734967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.0.201.189:35691)
0925 22:46:12.735211 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.735221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.735232 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:46:12.735266 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.735268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.735304 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.735376 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.735739 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.736399 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.737110 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.737328 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.737401 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.739319 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.739323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.739325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.739349 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.741113 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.741499 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.741509 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.741511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.741589 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.741829 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.741833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.741834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.742035 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.742126 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.742333 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.742450 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":110,"threads_started":1}
I20260925 22:46:12.743249 7633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.188@KRBTEST.COM for oryx/127.0.201.188@KRBTEST.COM
I20260925 22:46:12.750370 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.743739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:45327 (local address 127.0.0.1:33394)
0925 22:46:12.743826 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.743836 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.743918 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.744134 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.744137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.744281 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.744332 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.744337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.745079 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.745082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.745560 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.745566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.745990 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.745993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.746086 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.746518 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.746530 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.747569 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.749050 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.749054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.749056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.749226 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.749484 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.749489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.749491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.749559 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.749896 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.749900 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.749974 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.750076 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.750195 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":3}
I20260925 22:46:12.750402 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.743858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.0.201.188:45327)
0925 22:46:12.743962 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.743968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.743981 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:46:12.744009 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.744011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.744048 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.744127 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.744455 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.744961 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.745668 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.745890 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.745993 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.747716 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.747719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.747722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.747749 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.748919 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.749315 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.749320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.749322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.749398 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.749647 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.749650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.749651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.749795 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.749894 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.750115 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.750244 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":38}
I20260925 22:46:12.753412 7633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Bootstrap starting.
I20260925 22:46:12.755811 7633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:12.756495 7633 log.cc:824] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:12.758272 7633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: No bootstrap required, opened a new log
I20260925 22:46:12.761256 7106 heartbeater.cc:503] Master 127.0.201.190:33667 was elected leader, sending a full tablet report...
I20260925 22:46:12.761497 7633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } }
I20260925 22:46:12.761770 7633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:12.761828 7633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 990efe788dd84cec88dad8e7cefbb68b, State: Initialized, Role: FOLLOWER
I20260925 22:46:12.762260 7633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } }
I20260925 22:46:12.762737 7639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } } }
I20260925 22:46:12.762841 7639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:46:12.763480 7633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:46:12.767750 7650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:46:12.767828 7650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:46:12.770862 7487 master_runner.cc:428] Detected that this master 990efe788dd84cec88dad8e7cefbb68b is joining an existing cluster
I20260925 22:46:12.770954 7487 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.201.188:45327
I20260925 22:46:12.782624 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.773210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.0.201.189:35691)
0925 22:46:12.773312 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.773317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.773737 (+ 420us) server_negotiation.cc:407] Connection header received
0925 22:46:12.773905 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.773908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.773964 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.774022 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.774688 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.775462 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.776816 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.777145 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.777214 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.778670 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.778673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.778676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.778698 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.780470 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.781320 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.781325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.781327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.781413 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.781715 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.781718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.781720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.781946 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.782042 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.782366 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.782492 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 22:46:12.782624 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.773311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:35691 (local address 127.0.0.1:39378)
0925 22:46:12.773655 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.773670 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.773797 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.774166 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.774168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.774447 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.774568 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.774577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.775796 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.775799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.776701 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.776711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.777348 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.777351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.777552 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.778134 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.778149 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.778569 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.780596 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.780601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.780605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.780926 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.781515 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.781519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.781520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.781611 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.782029 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.782034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.782130 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.782309 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.782435 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":114,"thread_start_us":107,"threads_started":1}
I20260925 22:46:12.783393 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.773890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:45327 (local address 127.0.0.1:33396)
0925 22:46:12.774394 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.774407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.774515 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.774806 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.774809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.775020 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.775121 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.775128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.776139 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.776142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.777604 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.777614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.778514 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.778517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.778638 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.779086 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.779100 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.779498 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.781954 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.781960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.781964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.782177 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.782453 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.782456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.782458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.782541 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.782921 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.782924 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.783008 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.783103 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.783218 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":22,"thread_start_us":93,"threads_started":1}
I20260925 22:46:12.783413 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.773717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.0.201.188:45327)
0925 22:46:12.773806 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.773809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.774467 (+ 658us) server_negotiation.cc:407] Connection header received
0925 22:46:12.774602 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.774605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.774638 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.774681 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.775231 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.776024 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.777737 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.778386 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.778499 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.779617 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.779621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.779627 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.779653 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.781857 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.782272 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.782276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.782278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.782356 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.782624 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.782627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.782629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.782797 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.782856 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.783176 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.783278 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":384}
I20260925 22:46:12.785320 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 6977
I20260925 22:46:12.791348 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:44531
--local_ip_for_outbound_sockets=127.0.201.129
--tserver_master_addrs=127.0.201.190:33667
--webserver_port=43505
--webserver_interface=127.0.201.129
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab
--principal=oryx/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.201.190:33667,127.0.201.189:35691
--tserver_master_addrs=127.0.201.190:33667,127.0.201.189:35691,127.0.201.188:45327 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:46:12.800184 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.773218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.0.201.190:33667)
0925 22:46:12.773327 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.773332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.773592 (+ 260us) server_negotiation.cc:407] Connection header received
0925 22:46:12.773766 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.773769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.773814 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.773866 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.774638 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.775462 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.776781 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.777120 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.777215 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.796908 (+ 19693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.796912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.796914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.796940 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.798446 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.798919 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.798927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.798928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.799002 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.799293 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.799296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.799298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.799496 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.799606 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.799907 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.800017 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:46:12.800203 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.773156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:54340)
0925 22:46:12.773511 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.773526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.773656 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.774025 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.774028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.774326 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.774500 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.774509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.775589 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.775596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.776647 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.776657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.777345 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.777349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.777578 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.796308 (+ 18730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.796323 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.796706 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.798584 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.798588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.798590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.798807 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.799096 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.799101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.799102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.799190 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.799595 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.799598 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.799724 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.799850 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.800010 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":207,"spinlock_wait_cycles":6528,"thread_start_us":205,"threads_started":1}
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376371, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.190@KRBTEST.COM for oryx/127.0.201.188@KRBTEST.COM
I20260925 22:46:12.811594 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.804090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.0.201.188:45327)
0925 22:46:12.804398 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.804401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.804410 (+ 9us) server_negotiation.cc:407] Connection header received
0925 22:46:12.804462 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.804463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.804500 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.804585 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.805005 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.805589 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.806350 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.806573 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.806678 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.808629 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.808633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.808636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.808658 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.810022 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.810442 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.810448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.810450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.810530 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.810802 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.810807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.810809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.810973 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.811073 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.811339 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.811451 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260925 22:46:12.811594 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.804041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:45327 (local address 127.0.0.1:33400)
0925 22:46:12.804216 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.804228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.804286 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.804635 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.804638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.804814 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.804895 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.804901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.805691 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.805696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.806235 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.806242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.806680 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.806684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.806775 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.807216 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.807230 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.808497 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.810148 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.810151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.810153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.810347 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.810629 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.810631 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.810633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.810717 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.811074 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.811077 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.811176 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.811277 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.811421 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 22:46:12.812878 6914 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.201.188:45327: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b"
member_type: NON_VOTER
last_known_addr {
host: "127.0.201.188"
port: 45327
}
attrs {
promote: true
}
}
dest_uuid: "16a9d45479b44fefbc4803169474a9a5"
cas_config_opid_index: 5
I20260925 22:46:12.813445 6914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } }
I20260925 22:46:12.814530 7474 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 990efe788dd84cec88dad8e7cefbb68b (127.0.201.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.815972 7474 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.201.188:45327
I20260925 22:46:12.816143 7474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.816248 7474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:12.817071 7607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:12.818482 7607 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 FOLLOWER]: Refusing update from remote peer 16a9d45479b44fefbc4803169474a9a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:46:12.819154 7475 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:12.820243 7487 master.cc:537] Master@127.0.201.188:45327 shutting down...
I20260925 22:46:12.821157 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.814879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.0.201.189:35691)
0925 22:46:12.815001 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.815005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.816003 (+ 998us) server_negotiation.cc:407] Connection header received
0925 22:46:12.816243 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.816247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.816317 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.816412 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:12.816876 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.817608 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.819176 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.819811 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.819895 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.820055 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.820951 (+ 896us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.821040 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:46:12.821769 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.814642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:35691 (local address 127.0.0.1:39394)
0925 22:46:12.815932 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.815946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.816111 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.816652 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.816656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.816674 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:12.816756 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.816761 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.817794 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.817797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.819045 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.819053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.820232 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.820236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.820402 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.820819 (+ 417us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.821510 (+ 691us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.821605 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225}
I20260925 22:46:12.822533 7518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 LEARNER]: Refusing update from remote peer 16a9d45479b44fefbc4803169474a9a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:46:12.823042 7475 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 22:46:12.824846 7518 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 990efe788dd84cec88dad8e7cefbb68b (127.0.201.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.825759 7607 raft_consensus.cc:858] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } }
W20260925 22:46:12.825870 7607 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } }
I20260925 22:46:12.826727 7554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a9d45479b44fefbc4803169474a9a5. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.826835 7554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:46:12.826928 7554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.826994 7554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:46:12.827324 7607 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6daee056e6dd48e2a9d24fd3451b9460 (127.0.201.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } }
I20260925 22:46:12.829249 7665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } }
I20260925 22:46:12.829440 7665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 22:46:12.829591 7665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } }
I20260925 22:46:12.829692 7639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a9d45479b44fefbc4803169474a9a5. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } }
I20260925 22:46:12.829825 7639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 22:46:12.829945 7665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 22:46:12.831991 7475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: NON_VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: true } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.832118 7475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:12.833151 7474 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: attempting to promote NON_VOTER 6daee056e6dd48e2a9d24fd3451b9460 to VOTER
I20260925 22:46:12.833539 7474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } }
I20260925 22:46:12.834954 7518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 LEARNER]: Refusing update from remote peer 16a9d45479b44fefbc4803169474a9a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 22:46:12.835765 7487 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 22:46:12.835893 7661 mvcc.cc:205] Tried to move back new op lower bound from 7333381617675616256 to 7333381617496748032. Current Snapshot: MvccSnapshot[applied={T|T < 7333381617534676992}]
I20260925 22:46:12.836006 7487 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 22:46:12.836088 7487 tablet_replica.cc:330] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: stopping tablet replica
I20260925 22:46:12.836858 7475 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260925 22:46:12.837556 6899 consensus_peers.cc:487] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 -> Peer 990efe788dd84cec88dad8e7cefbb68b (127.0.201.188:45327): Couldn't send request to peer 990efe788dd84cec88dad8e7cefbb68b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 22:46:12.838418 7475 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6daee056e6dd48e2a9d24fd3451b9460 (127.0.201.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.838899 7518 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6daee056e6dd48e2a9d24fd3451b9460 (127.0.201.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.840085 7666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.840135 7475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.840202 7666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:12.840247 7475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:12.840549 7662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.840658 7662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:46:12.859159 7487 master.cc:552] Master@127.0.201.188:45327 shutdown complete.
I20260925 22:46:12.859238 7487 master_runner.cc:305] Clearing existing system tablet
I20260925 22:46:12.859468 7487 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:12.859539 7487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:12.860996 7487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:12.861912 7671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.862119 7487 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:12.862246 7487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 22:46:12.862287 7487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal
uuid: "990efe788dd84cec88dad8e7cefbb68b"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "b73ea72a2fcfba4dc821d8b6fa1b9152"
server_key_iv: "d6663f036f5217b417f404fc2e008ba8"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.863794 7487 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:12.865860 7487 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:46:12.865970 7487 log.cc:1196] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 22:46:12.866218 7487 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Deleting consensus metadata
I20260925 22:46:12.866403 7487 master_runner.cc:315] Copying system tablet from 127.0.201.190:33667
I20260925 22:46:12.867484 7487 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.201.190:33667
I20260925 22:46:12.878620 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.868129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.0.201.190:33667)
0925 22:46:12.868258 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.868264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.871146 (+ 2882us) server_negotiation.cc:407] Connection header received
0925 22:46:12.871297 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.871300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.871352 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.871544 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.872206 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.872738 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.873555 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.873785 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.873892 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.875460 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.875464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.875467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.875490 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.877001 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.877453 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.877457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.877459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.877537 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.877805 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.877810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.877812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.877986 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.878085 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.878342 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.878456 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:46:12.878628 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.868077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:54350)
0925 22:46:12.871041 (+ 2964us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:12.871053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:12.871191 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:12.871581 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:12.871584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:12.871929 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:12.872091 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.872097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.872845 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.872851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.873431 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:12.873438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.873897 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.873900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:12.873997 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.874999 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:12.875011 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:12.875316 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:12.877147 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.877151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.877153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.877353 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.877627 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:12.877631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:12.877633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:12.877716 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:12.878086 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:12.878089 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:12.878188 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:46:12.878293 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 22:46:12.878434 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2893,"thread_start_us":80,"threads_started":1}
I20260925 22:46:12.879352 6935 tablet_copy_service.cc:140] P 16a9d45479b44fefbc4803169474a9a5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 990efe788dd84cec88dad8e7cefbb68b ({username='oryx', principal='oryx/127.0.201.188@KRBTEST.COM'} at 127.0.0.1:54350)
I20260925 22:46:12.879467 6935 tablet_copy_service.cc:161] P 16a9d45479b44fefbc4803169474a9a5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 990efe788dd84cec88dad8e7cefbb68b at {username='oryx', principal='oryx/127.0.201.188@KRBTEST.COM'} at 127.0.0.1:54350: session id = 990efe788dd84cec88dad8e7cefbb68b-00000000000000000000000000000000
I20260925 22:46:12.880434 6935 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:46:12.881502 7487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:46:12.883230 7487 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: tablet copy: Starting download of 0 data blocks...
I20260925 22:46:12.883438 7487 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: tablet copy: Starting download of 1 WAL segments...
I20260925 22:46:12.884773 7487 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:46:12.886185 6935 tablet_copy_service.cc:342] P 16a9d45479b44fefbc4803169474a9a5: Request end of tablet copy session 990efe788dd84cec88dad8e7cefbb68b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.201.188@KRBTEST.COM'} at 127.0.0.1:54350
I20260925 22:46:12.886278 6935 tablet_copy_service.cc:433] P 16a9d45479b44fefbc4803169474a9a5: ending tablet copy session 990efe788dd84cec88dad8e7cefbb68b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 990efe788dd84cec88dad8e7cefbb68b
I20260925 22:46:12.892321 7487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:12.893864 7678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.893909 7681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.894352 7679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:12.895833 7487 server_base.cc:1034] running on GCE node
I20260925 22:46:12.896013 7487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:12.896308 7487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:12.897465 7487 hybrid_clock.cc:648] HybridClock initialized: now 1790376372897475 us; error 45 us; skew 500 ppm
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:12.898900 7487 init.cc:377] Logged in from keytab as oryx/127.0.201.188@KRBTEST.COM (short username oryx)
I20260925 22:46:12.899359 7487 webserver.cc:460] Webserver started at http://127.0.201.188:32845/ using document root <none> and password file <none>
I20260925 22:46:12.899550 7487 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:12.899622 7487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:12.900790 7487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:12.901688 7688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.901906 7487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:12.902001 7487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal
uuid: "990efe788dd84cec88dad8e7cefbb68b"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "b73ea72a2fcfba4dc821d8b6fa1b9152"
server_key_iv: "d6663f036f5217b417f404fc2e008ba8"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.902079 7487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:12.914712 7487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:12.916664 7487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:12.921537 7487 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.188:45327
I20260925 22:46:12.922014 7487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
I20260925 22:46:12.922397 7740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.188:45327 every 8 connection(s)
I20260925 22:46:12.923491 7741 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:46:12.924456 7741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Bootstrap starting.
I20260925 22:46:12.931511 7741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:46:12.931799 7741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Bootstrap complete.
I20260925 22:46:12.932317 7741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } }
I20260925 22:46:12.932449 7741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 990efe788dd84cec88dad8e7cefbb68b, State: Initialized, Role: LEARNER
I20260925 22:46:12.932559 7741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } }
I20260925 22:46:12.933244 7741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:12.936939 7755 catalog_manager.cc:1259] Loaded cluster ID: df6895d752294032b572668a61c7b7c7
I20260925 22:46:12.936990 7755 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: loading cluster ID for follower catalog manager: success
I20260925 22:46:12.935010 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:12.937477 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: This master's current role is: LEARNER
I20260925 22:46:12.938786 7755 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: acquiring CA information for follower catalog manager: success
I20260925 22:46:12.940203 7755 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:46:12.944939 7659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:12.945250 7659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:12.945317 7659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:12.951143 7659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:12.951215 7659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:12.951298 7659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:12.959369 7659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab
--principal=oryx/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:44531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=43505
--tserver_master_addrs=127.0.201.190:33667,127.0.201.189:35691,127.0.201.188:45327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:12.960597 7659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:12.961880 7659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:12.973591 7761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.973786 7760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.973786 7763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:12.974100 7659 server_base.cc:1034] running on GCE node
I20260925 22:46:12.974478 7659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:12.975222 7659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:12.976251 7240 heartbeater.cc:503] Master 127.0.201.190:33667 was elected leader, sending a full tablet report...
I20260925 22:46:12.976404 7659 hybrid_clock.cc:648] HybridClock initialized: now 1790376372976388 us; error 33 us; skew 500 ppm
Sep 25 22:46:12 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:12.979246 7659 init.cc:377] Logged in from keytab as oryx/127.0.201.129@KRBTEST.COM (short username oryx)
I20260925 22:46:12.980113 7659 webserver.cc:460] Webserver started at http://127.0.201.129:43505/ using document root <none> and password file <none>
I20260925 22:46:12.980649 7659 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:12.980717 7659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:12.984424 7659 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:12.986320 7770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.987474 7659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:12.987723 7659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "427fbaeacb8b4c66ad3417551bc7174c"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "19e29733aaafd5ff8d45a47d425fe68e"
server_key_iv: "bf6685dfbf6f6101a118a2b7da761710"
server_key_version: "encryptionkey@0"
I20260925 22:46:12.988229 7659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:13.001793 7659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:13.005134 7659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:13.005362 7659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:13.005945 7659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:13.006788 7659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:13.006834 7659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.006896 7659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:13.006938 7659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.016444 7659 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:44531
I20260925 22:46:13.016505 7883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:44531 every 8 connection(s)
I20260925 22:46:13.017433 7659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:13.022485 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 7659
I20260925 22:46:13.022689 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 7111
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.188@KRBTEST.COM
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.189@KRBTEST.COM
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:13.031930 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:32939
--local_ip_for_outbound_sockets=127.0.201.130
--tserver_master_addrs=127.0.201.190:33667
--webserver_port=33731
--webserver_interface=127.0.201.130
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab
--principal=oryx/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.201.190:33667,127.0.201.189:35691
--tserver_master_addrs=127.0.201.190:33667,127.0.201.189:35691,127.0.201.188:45327 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:46:13.035981 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.019349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:43437 (local address 127.0.201.190:33667)
0925 22:46:13.019473 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.019477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.020453 (+ 976us) server_negotiation.cc:407] Connection header received
0925 22:46:13.021541 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.021544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.021602 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.021653 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.023443 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.024256 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.026311 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.026660 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.026738 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.031304 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.031308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.031311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.031338 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.033705 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.034395 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.034398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.034400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.034477 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.034783 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.034786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.034788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.034998 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.035062 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.035649 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.035802 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:46:13.036068 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.019189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:51451 (local address 127.0.201.189:35691)
0925 22:46:13.019340 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.019345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.020645 (+ 1300us) server_negotiation.cc:407] Connection header received
0925 22:46:13.021522 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.021525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.021584 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.021660 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.023299 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.024216 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.026304 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.026801 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.026926 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.031288 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.031294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.031300 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.031332 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.033502 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.034122 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.034127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.034132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.034219 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.034659 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.034662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.034664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.034887 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.034976 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.035820 (+ 844us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.035928 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:46:13.036919 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.020773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:47139 (local address 127.0.201.188:45327)
0925 22:46:13.021005 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.021009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.021021 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:46:13.021625 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.021631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.021685 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.021762 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.023440 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.024496 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.026311 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.026665 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.026749 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.031304 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.031308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.031311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.031338 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.033705 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.034339 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.034345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.034349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.034439 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.034738 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.034744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.034749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.035000 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.035095 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.036510 (+ 1415us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.036656 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
I20260925 22:46:13.037578 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.019594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:35691 (local address 127.0.201.129:51451)
0925 22:46:13.020551 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.020562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.021324 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.021829 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.021836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.022315 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:13.023136 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.023155 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.024378 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.024385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.026165 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.026178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.027081 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.027087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.027307 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.028138 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:13.028159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:13.031122 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:13.033610 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.033616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.033630 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.033998 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.034321 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.034324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.034328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.034559 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.035030 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:13.035038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:13.035348 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.035579 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.036371 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":371,"spinlock_wait_cycles":30848,"thread_start_us":109,"threads_started":1}
I20260925 22:46:13.037578 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.019594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.129:43437)
0925 22:46:13.020206 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.020271 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.021327 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.021832 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.021839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.022317 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:13.023299 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.023303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.024397 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.024400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.026176 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.026186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.026909 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.026913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.027236 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.028507 (+ 1271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:13.028520 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:13.031111 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:13.033809 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.033812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.033815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.034200 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.034593 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.034599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.034602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.034682 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.035124 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:13.035127 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:13.035348 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.036807 (+ 1459us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.036954 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":930,"spinlock_wait_cycles":26240,"thread_start_us":711,"threads_started":1}
I20260925 22:46:13.038265 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.020648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:45327 (local address 127.0.201.129:47139)
0925 22:46:13.020934 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.020947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.021327 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.021877 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.021879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.022317 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:13.023293 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.023301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.024612 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.024615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.026176 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.026186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.026909 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.026913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.027237 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.027977 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:13.028001 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:13.031111 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:13.033836 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.033842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.033845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.034220 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.034545 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.034548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.034551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.034638 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.035139 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:13.035145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:13.035348 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.035579 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.035962 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":772,"spinlock_wait_cycles":23936,"thread_start_us":70,"threads_started":1}
I20260925 22:46:13.038949 7886 heartbeater.cc:347] Connected to a master server at 127.0.201.189:35691
I20260925 22:46:13.039041 7885 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33667
I20260925 22:46:13.039208 7886 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:13.039230 7885 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:13.039764 7885 heartbeater.cc:511] Master 127.0.201.190:33667 requested a full tablet report, sending...
I20260925 22:46:13.039765 7886 heartbeater.cc:511] Master 127.0.201.189:35691 requested a full tablet report, sending...
I20260925 22:46:13.040587 7884 heartbeater.cc:347] Connected to a master server at 127.0.201.188:45327
I20260925 22:46:13.040663 7884 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:13.040800 7884 heartbeater.cc:511] Master 127.0.201.188:45327 requested a full tablet report, sending...
I20260925 22:46:13.041158 6914 ts_manager.cc:194] Re-registered known tserver with Master: 427fbaeacb8b4c66ad3417551bc7174c (127.0.201.129:44531)
I20260925 22:46:13.041288 7506 ts_manager.cc:194] Registered new tserver with Master: 427fbaeacb8b4c66ad3417551bc7174c (127.0.201.129:44531)
I20260925 22:46:13.042033 6914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:43437
I20260925 22:46:13.043249 7705 ts_manager.cc:194] Registered new tserver with Master: 427fbaeacb8b4c66ad3417551bc7174c (127.0.201.129:44531)
W20260925 22:46:13.146371 7892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:13.146669 7892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:13.146747 7892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:13.150801 7892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:13.150897 7892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:13.150988 7892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:13.156337 7892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab
--principal=oryx/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:32939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=33731
--tserver_master_addrs=127.0.201.190:33667,127.0.201.189:35691,127.0.201.188:45327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:13.157457 7892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:13.158473 7892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:13.168277 7898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:13.168293 7897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:13.168471 7892 server_base.cc:1034] running on GCE node
W20260925 22:46:13.168569 7900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:13.169656 7892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:13.170372 7892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:13.171568 7892 hybrid_clock.cc:648] HybridClock initialized: now 1790376373171532 us; error 47 us; skew 500 ppm
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:13.174381 7892 init.cc:377] Logged in from keytab as oryx/127.0.201.130@KRBTEST.COM (short username oryx)
I20260925 22:46:13.175271 7892 webserver.cc:460] Webserver started at http://127.0.201.130:33731/ using document root <none> and password file <none>
I20260925 22:46:13.175873 7892 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:13.175947 7892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:13.179564 7892 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:13.181614 7907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.182773 7892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:13.182898 7892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "7cd22e79ff714a928cd1116f0261e49e"
format_stamp: "Formatted at 2026-09-25 22:46:11 on dist-test-slave-vdtf"
server_key: "3cfbf779d6fe265fe6a995cce488d2dc"
server_key_iv: "b86a7e9d10861e055b5c4a14638649e2"
server_key_version: "encryptionkey@0"
I20260925 22:46:13.183429 7892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:13.189018 7374 heartbeater.cc:503] Master 127.0.201.190:33667 was elected leader, sending a full tablet report...
I20260925 22:46:13.195242 7892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:13.198495 7892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:13.198686 7892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:13.199265 7892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:13.200161 7892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:13.200205 7892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.200266 7892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:13.200297 7892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.209345 7892 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:32939
I20260925 22:46:13.209446 8020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:32939 every 8 connection(s)
I20260925 22:46:13.210353 7892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:13.219434 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 7892
I20260925 22:46:13.219643 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 7245
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.188@KRBTEST.COM
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.189@KRBTEST.COM
I20260925 22:46:13.226948 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.213827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:57917 (local address 127.0.201.188:45327)
0925 22:46:13.213966 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.213970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.214064 (+ 94us) server_negotiation.cc:407] Connection header received
0925 22:46:13.214340 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.214344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.214399 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.214454 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.216123 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.216921 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.218303 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.218708 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.218821 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.222404 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.222407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.222410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.222438 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.224884 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.225439 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.225446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.225449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.225531 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.225829 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.225833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.225834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.226064 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.226131 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.226659 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.226779 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:46:13.226948 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.212251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:46801 (local address 127.0.201.189:35691)
0925 22:46:13.212393 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.212398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.213464 (+ 1066us) server_negotiation.cc:407] Connection header received
0925 22:46:13.214332 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.214336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.214391 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.214488 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.216094 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.216606 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.217780 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.217997 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.218114 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.223382 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.223387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.223390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.223420 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.225188 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.225647 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.225651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.225652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.225721 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.226025 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.226029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.226030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.226250 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.226313 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.226659 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.226778 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:46:13.227033 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.212154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:39171 (local address 127.0.201.190:33667)
0925 22:46:13.212319 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.212324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.213253 (+ 929us) server_negotiation.cc:407] Connection header received
0925 22:46:13.214340 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.214344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.214404 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.214454 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.216121 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.216921 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.218323 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.218555 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.218611 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.222384 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.222391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.222397 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.222431 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.224465 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.225110 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.225117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.225122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.225210 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.225625 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.225631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.225636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.225881 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.225956 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.226660 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.226784 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:46:13.228041 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:36793
--local_ip_for_outbound_sockets=127.0.201.131
--tserver_master_addrs=127.0.201.190:33667
--webserver_port=37063
--webserver_interface=127.0.201.131
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab
--principal=oryx/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.201.190:33667,127.0.201.189:35691
--tserver_master_addrs=127.0.201.190:33667,127.0.201.189:35691,127.0.201.188:45327 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:46:13.228713 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.213706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:45327 (local address 127.0.201.130:57917)
0925 22:46:13.213982 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.213994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.214124 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.214664 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.214671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.215251 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:13.215962 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.215977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.217063 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.217066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.218187 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.218197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.218822 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.218825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.218973 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.219533 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:13.219547 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:13.222221 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:13.225066 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.225069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.225071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.225316 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.225645 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.225648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.225650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.225728 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.226198 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:13.226202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:13.226391 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.227168 (+ 777us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.227732 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":490,"spinlock_wait_cycles":14976,"thread_start_us":132,"threads_started":1}
I20260925 22:46:13.228713 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.212577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:35691 (local address 127.0.201.130:46801)
0925 22:46:13.213379 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.213392 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.214121 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.214663 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.214670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.215426 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:13.215961 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.215974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.216741 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.216749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.217649 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.217657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.218144 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.218147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.218348 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.219061 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:13.219084 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:13.223207 (+ 4123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:13.225304 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.225307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.225309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.225540 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.225829 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.225832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.225834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.225923 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.226380 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:13.226383 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:13.226450 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.227168 (+ 718us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.227654 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":289,"spinlock_wait_cycles":13440,"thread_start_us":158,"threads_started":1}
I20260925 22:46:13.228713 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.212577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.130:39171)
0925 22:46:13.213084 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.213135 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.214124 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.214664 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.214671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.215251 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:13.215962 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.215977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.217063 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.217066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.218187 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.218197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.218797 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.218801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.218993 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.219641 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:13.219659 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:13.222196 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:13.224596 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.224604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.224622 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.224979 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.225333 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.225339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.225344 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.225518 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.226050 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:13.226059 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:13.226388 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.227183 (+ 795us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.227578 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":342,"spinlock_wait_cycles":16768,"thread_start_us":567,"threads_started":1}
I20260925 22:46:13.229985 8023 heartbeater.cc:347] Connected to a master server at 127.0.201.189:35691
I20260925 22:46:13.230089 8022 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33667
I20260925 22:46:13.230198 8022 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:13.230243 8021 heartbeater.cc:347] Connected to a master server at 127.0.201.188:45327
I20260925 22:46:13.230197 8023 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:13.230315 8021 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:13.230777 8023 heartbeater.cc:511] Master 127.0.201.189:35691 requested a full tablet report, sending...
I20260925 22:46:13.230777 8021 heartbeater.cc:511] Master 127.0.201.188:45327 requested a full tablet report, sending...
I20260925 22:46:13.231371 8022 heartbeater.cc:511] Master 127.0.201.190:33667 requested a full tablet report, sending...
I20260925 22:46:13.231986 7705 ts_manager.cc:194] Registered new tserver with Master: 7cd22e79ff714a928cd1116f0261e49e (127.0.201.130:32939)
I20260925 22:46:13.231988 7506 ts_manager.cc:194] Registered new tserver with Master: 7cd22e79ff714a928cd1116f0261e49e (127.0.201.130:32939)
I20260925 22:46:13.232733 6914 ts_manager.cc:194] Re-registered known tserver with Master: 7cd22e79ff714a928cd1116f0261e49e (127.0.201.130:32939)
I20260925 22:46:13.233619 6914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:39171
W20260925 22:46:13.340227 8028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:13.340575 8028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:13.340652 8028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:13.344712 8028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:13.344767 8028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:13.344853 8028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:13.349977 8028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab
--principal=oryx/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:36793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=37063
--tserver_master_addrs=127.0.201.190:33667,127.0.201.189:35691,127.0.201.188:45327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:13.350986 8028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:13.352049 8028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:13.361835 8034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:13.361791 8037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:13.361791 8035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:13.362035 8028 server_base.cc:1034] running on GCE node
I20260925 22:46:13.362406 8028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:13.363005 8028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:13.364182 8028 hybrid_clock.cc:648] HybridClock initialized: now 1790376373364171 us; error 22 us; skew 500 ppm
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:13.366852 8028 init.cc:377] Logged in from keytab as oryx/127.0.201.131@KRBTEST.COM (short username oryx)
I20260925 22:46:13.368112 8028 webserver.cc:460] Webserver started at http://127.0.201.131:37063/ using document root <none> and password file <none>
I20260925 22:46:13.368695 8028 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:13.368803 8028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:13.372975 8028 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:46:13.374956 8044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.376392 8028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:13.376521 8028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "f4af0e5f397c45908604223c8fff2b4f"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-vdtf"
server_key: "cc1a9c7fb3bcd2f1c9ee7ef40e70b1f5"
server_key_iv: "efa1dac04bc3e892dadee5dfc9c2c7ad"
server_key_version: "encryptionkey@0"
I20260925 22:46:13.377081 8028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:13.416757 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.413237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:45327 (local address 127.0.0.1:33402)
0925 22:46:13.413511 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.413528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.413681 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.413981 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.413984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.414000 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:13.414109 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.414115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.414848 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.414851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.415622 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.415630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.416354 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.416357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.416451 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.416474 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.416538 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.416617 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260925 22:46:13.416756 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.413364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.0.201.188:45327)
0925 22:46:13.413502 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.413506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.413615 (+ 109us) server_negotiation.cc:407] Connection header received
0925 22:46:13.413787 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.413790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.413851 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.413972 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:13.414212 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.414730 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.415791 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.416227 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.416349 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.416525 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.416562 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.416620 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:46:13.417421 7715 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 22:46:13.417681 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a9d45479b44fefbc4803169474a9a5. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:13.417824 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: This master's current role is: LEARNER
I20260925 22:46:13.450409 8028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:13.453482 8028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:13.453686 8028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:13.454288 8028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:13.455200 8028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:13.455279 8028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.455348 8028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:13.455370 8028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.464475 8028 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:36793
I20260925 22:46:13.464499 8158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:36793 every 8 connection(s)
I20260925 22:46:13.465430 8028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:13.467320 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 8028
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.188@KRBTEST.COM
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
Sep 25 22:46:13 dist-test-slave-vdtf krb5kdc[6870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.189@KRBTEST.COM
I20260925 22:46:13.482196 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.469978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.0.201.190:33667)
0925 22:46:13.470195 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.470199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.471996 (+ 1797us) server_negotiation.cc:407] Connection header received
0925 22:46:13.472043 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.472048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.472100 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.472176 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.473042 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.473790 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.474879 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.475210 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.475315 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.477975 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.477979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.477981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.478008 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.479877 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.480425 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.480432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.480437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.480524 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.480884 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.480890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.480895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.481114 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.481253 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.481789 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.481978 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
I20260925 22:46:13.485559 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.467472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:56375 (local address 127.0.201.189:35691)
0925 22:46:13.467762 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.467767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.469658 (+ 1891us) server_negotiation.cc:407] Connection header received
0925 22:46:13.469952 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.469956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.470017 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.470078 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.473543 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.474254 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.475329 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.475630 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.475745 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.481685 (+ 5940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.481688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.481691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.481735 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.483662 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.484323 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.484326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.484328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.484416 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.484710 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.484713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.484715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.484901 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.484980 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.485293 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.485408 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260925 22:46:13.485559 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.467298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:36445 (local address 127.0.201.190:33667)
0925 22:46:13.467614 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.467621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.468482 (+ 861us) server_negotiation.cc:407] Connection header received
0925 22:46:13.469961 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.469964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.470014 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.470072 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.472113 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.472888 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.476028 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.476345 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.476421 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.480095 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.480098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.480100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.480123 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.483144 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.483829 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.483832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.483834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.483936 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.484322 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.484325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.484328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.484534 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.484609 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.485294 (+ 685us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.485403 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260925 22:46:13.487162 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.467691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.131:36445)
0925 22:46:13.468309 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.468367 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.469725 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.470471 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.470483 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.471105 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:13.471936 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.471957 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.473063 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.473066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.474451 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.474461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.476660 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.476663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.476734 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.478252 (+ 1518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:13.478275 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:13.479923 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:13.483319 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.483326 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.483342 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.483719 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.484044 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.484046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.484048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.484209 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.484673 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:13.484682 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:13.485017 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.485703 (+ 686us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.486091 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":140,"spinlock_wait_cycles":12928,"thread_start_us":699,"threads_started":1}
I20260925 22:46:13.488042 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.469234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.0.201.189:35691)
0925 22:46:13.469491 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.469497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.470954 (+ 1457us) server_negotiation.cc:407] Connection header received
0925 22:46:13.470994 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.470996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.471046 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.471127 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.472159 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.473122 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.474277 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.474634 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.474724 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.477957 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.477961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.477964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.477989 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.485252 (+ 7263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.486220 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.486223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.486225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.486306 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.486579 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.486582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.486584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.486768 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.486883 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.487767 (+ 884us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.487883 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260925 22:46:13.488430 8160 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33667
I20260925 22:46:13.488670 8160 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:13.488967 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.469020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:57349 (local address 127.0.201.188:45327)
0925 22:46:13.469144 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.469149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.470671 (+ 1522us) server_negotiation.cc:407] Connection header received
0925 22:46:13.470839 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.470846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.470904 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.470960 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.472094 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.472942 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.475975 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.476302 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.476401 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.480942 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.480947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.480949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.480971 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.482776 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.487622 (+ 4846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.487627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.487629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.487737 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.488081 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.488084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.488086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.488291 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.488369 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.488697 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.488811 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:46:13.485911 6914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:46:13.489238 8160 heartbeater.cc:511] Master 127.0.201.190:33667 requested a full tablet report, sending...
I20260925 22:46:13.489408 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.468918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:45327 (local address 127.0.201.131:57349)
0925 22:46:13.470583 (+ 1665us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.470600 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.470731 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.471118 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.471120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.471284 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:13.471935 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.471958 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.473074 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.473078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.475884 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.475892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.476518 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.476521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.476732 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.477464 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:13.477487 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:13.480801 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:13.484348 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.484352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.484356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.487509 (+ 3153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.487851 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.487866 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.487868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.487961 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.488425 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:13.488429 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:13.488521 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.489080 (+ 559us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.489218 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1596,"mutex_wait_us":1326,"spinlock_wait_cycles":3584,"thread_start_us":84,"threads_started":1}
I20260925 22:46:13.490266 6915 ts_manager.cc:194] Re-registered known tserver with Master: f4af0e5f397c45908604223c8fff2b4f (127.0.201.131:36793)
I20260925 22:46:13.490372 8159 heartbeater.cc:347] Connected to a master server at 127.0.201.188:45327
I20260925 22:46:13.490458 8159 heartbeater.cc:464] Registering TS with master...
W20260925 22:46:13.490572 6914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:13.490612 8159 heartbeater.cc:511] Master 127.0.201.188:45327 requested a full tablet report, sending...
I20260925 22:46:13.491006 6915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:36445
I20260925 22:46:13.491362 7705 ts_manager.cc:194] Registered new tserver with Master: f4af0e5f397c45908604223c8fff2b4f (127.0.201.131:36793)
I20260925 22:46:13.491585 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.470315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.0.201.188:45327)
0925 22:46:13.476498 (+ 6183us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.476503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.476517 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:46:13.476557 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.476561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.476609 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.476695 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.477542 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.478203 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.479485 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.479894 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.479973 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.482003 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.482009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.482012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.482040 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.486279 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.489998 (+ 3719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.490001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.490003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.490094 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.490411 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.490415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.490416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.490601 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.490717 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.491317 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.491438 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6075,"thread_start_us":82,"threads_started":1}
I20260925 22:46:13.493158 8168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.467713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:35691 (local address 127.0.201.131:56375)
0925 22:46:13.469551 (+ 1838us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.469564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.469738 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.470472 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.470483 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.471104 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:13.473416 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.473425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.474382 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.474385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.475211 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.475219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.476486 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.476495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.476807 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.478851 (+ 2044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:13.478887 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:13.481541 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:13.483989 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.483993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.483995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.484216 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.484524 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.484527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.484529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.484609 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.485034 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:13.485036 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:13.485107 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.485703 (+ 596us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.486447 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1767,"mutex_wait_us":697,"spinlock_wait_cycles":27776,"thread_start_us":164,"threads_started":1}
I20260925 22:46:13.493695 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: NON_VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: true } } }
I20260925 22:46:13.493814 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:13.495172 8032 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: attempting to promote NON_VOTER 990efe788dd84cec88dad8e7cefbb68b to VOTER
I20260925 22:46:13.495540 8032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } }
I20260925 22:46:13.495667 8161 heartbeater.cc:347] Connected to a master server at 127.0.201.189:35691
I20260925 22:46:13.495767 8161 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:13.495962 8161 heartbeater.cc:511] Master 127.0.201.189:35691 requested a full tablet report, sending...
I20260925 22:46:13.496672 7506 ts_manager.cc:194] Registered new tserver with Master: f4af0e5f397c45908604223c8fff2b4f (127.0.201.131:36793)
I20260925 22:46:13.496788 7715 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 LEARNER]: Refusing update from remote peer 16a9d45479b44fefbc4803169474a9a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:46:13.496794 7518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Refusing update from remote peer 16a9d45479b44fefbc4803169474a9a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:46:13.497165 8177 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 22:46:13.497320 8178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 22:46:13.498971 8178 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 990efe788dd84cec88dad8e7cefbb68b (127.0.201.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:13.499285 7518 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 990efe788dd84cec88dad8e7cefbb68b (127.0.201.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:13.501194 7715 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 990efe788dd84cec88dad8e7cefbb68b (127.0.201.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:13.501654 8186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:13.501767 8186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:46:13.502789 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:13.502907 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:13.502846 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:13.502991 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:46:13.503423 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:13.503527 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:13.503695 8178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16a9d45479b44fefbc4803169474a9a5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:13.503779 8178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:13.511924 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.506252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:32939 (local address 127.0.0.1:34400)
0925 22:46:13.506457 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.506473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.506551 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.507571 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.507573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.507591 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:13.507723 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.507730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.509185 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.509188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.510447 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.510459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.511502 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.511506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.511595 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.511635 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.511705 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.511778 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":70}
I20260925 22:46:13.512131 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.506544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44531 (local address 127.0.0.1:55588)
0925 22:46:13.506762 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.506774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.506849 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.507755 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.507757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.507776 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:13.507877 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.507883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.509069 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.509072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.510459 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.510466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.511721 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.511726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.511819 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.511843 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.511922 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.511986 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":82,"thread_start_us":105,"threads_started":1}
I20260925 22:46:13.512216 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.506804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.0.201.129:44531)
0925 22:46:13.507046 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.507061 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.507085 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:46:13.507165 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.507171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.507408 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.507572 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:13.508014 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.508947 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.510586 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.511564 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.511666 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.511839 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.511955 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.512062 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260925 22:46:13.512596 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.506528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.0.201.130:32939)
0925 22:46:13.507092 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.507105 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.507126 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:46:13.507201 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.507207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.507403 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.507872 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:13.508004 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.509084 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.510586 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.511369 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.512176 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.512329 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.512378 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.512472 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":123,"threads_started":1}
I20260925 22:46:13.514514 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.507516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36793 (local address 127.0.0.1:39192)
0925 22:46:13.507914 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.508626 (+ 712us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.508706 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.509357 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.509360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.509377 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:13.509481 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.509488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.510426 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.510430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.512936 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.512946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.513995 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.513999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.514082 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.514104 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.514151 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.514397 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260925 22:46:13.515177 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.507882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.0.201.131:36793)
0925 22:46:13.508943 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.508953 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.508972 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:46:13.509031 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.509035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.509219 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.509382 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:13.509571 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.510292 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.513109 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.513847 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.514757 (+ 910us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.514891 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.514950 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.515037 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":106,"threads_started":1}
I20260925 22:46:13.515179 7818 tablet_service.cc:1467] Processing CreateTablet for tablet a6eae66701894675bdd63455f04d3a86 (DEFAULT_TABLE table=test-table [id=16f8bfa3708742adbb2e2a408268e602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:13.515146 7955 tablet_service.cc:1467] Processing CreateTablet for tablet a6eae66701894675bdd63455f04d3a86 (DEFAULT_TABLE table=test-table [id=16f8bfa3708742adbb2e2a408268e602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:13.516193 7955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eae66701894675bdd63455f04d3a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:13.516216 7818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eae66701894675bdd63455f04d3a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:13.517735 8093 tablet_service.cc:1467] Processing CreateTablet for tablet a6eae66701894675bdd63455f04d3a86 (DEFAULT_TABLE table=test-table [id=16f8bfa3708742adbb2e2a408268e602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:13.518643 8093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eae66701894675bdd63455f04d3a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:13.522588 8193 tablet_bootstrap.cc:492] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: Bootstrap starting.
I20260925 22:46:13.524041 8194 tablet_bootstrap.cc:492] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: Bootstrap starting.
I20260925 22:46:13.525475 8193 tablet_bootstrap.cc:654] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:13.526090 8194 tablet_bootstrap.cc:654] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:13.526343 8193 log.cc:824] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:13.526402 8195 tablet_bootstrap.cc:492] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: Bootstrap starting.
I20260925 22:46:13.526937 8194 log.cc:824] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:13.528321 8193 tablet_bootstrap.cc:492] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: No bootstrap required, opened a new log
I20260925 22:46:13.528506 8193 ts_tablet_manager.cc:1397] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:46:13.528517 8195 tablet_bootstrap.cc:654] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:13.528873 8194 tablet_bootstrap.cc:492] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: No bootstrap required, opened a new log
I20260925 22:46:13.529042 8194 ts_tablet_manager.cc:1397] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:46:13.529294 8195 log.cc:824] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:13.531055 8195 tablet_bootstrap.cc:492] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: No bootstrap required, opened a new log
I20260925 22:46:13.531245 8195 ts_tablet_manager.cc:1397] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:46:13.533186 8193 raft_consensus.cc:348] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } } peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } }
I20260925 22:46:13.533442 8193 raft_consensus.cc:374] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:13.533497 8193 raft_consensus.cc:729] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd22e79ff714a928cd1116f0261e49e, State: Initialized, Role: FOLLOWER
I20260925 22:46:13.533471 8194 raft_consensus.cc:348] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } } peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } }
I20260925 22:46:13.533699 8194 raft_consensus.cc:374] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:13.533762 8194 raft_consensus.cc:729] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427fbaeacb8b4c66ad3417551bc7174c, State: Initialized, Role: FOLLOWER
I20260925 22:46:13.533982 8193 consensus_queue.cc:260] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } } peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } }
I20260925 22:46:13.534119 8194 consensus_queue.cc:260] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } } peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } }
I20260925 22:46:13.535221 8193 ts_tablet_manager.cc:1428] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:46:13.535221 8194 ts_tablet_manager.cc:1428] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:46:13.535347 8195 raft_consensus.cc:348] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } } peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } }
I20260925 22:46:13.535595 8195 raft_consensus.cc:374] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:13.535600 7885 heartbeater.cc:503] Master 127.0.201.190:33667 was elected leader, sending a full tablet report...
I20260925 22:46:13.535645 8195 raft_consensus.cc:729] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4af0e5f397c45908604223c8fff2b4f, State: Initialized, Role: FOLLOWER
I20260925 22:46:13.536070 8195 consensus_queue.cc:260] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } } peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } }
I20260925 22:46:13.536499 8022 heartbeater.cc:503] Master 127.0.201.190:33667 was elected leader, sending a full tablet report...
I20260925 22:46:13.536983 8195 ts_tablet_manager.cc:1428] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:46:13.537806 8160 heartbeater.cc:503] Master 127.0.201.190:33667 was elected leader, sending a full tablet report...
I20260925 22:46:13.543402 8201 raft_consensus.cc:482] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:13.543546 8201 raft_consensus.cc:504] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } } peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } }
I20260925 22:46:13.545558 8201 leader_election.cc:288] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cd22e79ff714a928cd1116f0261e49e (127.0.201.130:32939), 427fbaeacb8b4c66ad3417551bc7174c (127.0.201.129:44531)
I20260925 22:46:13.548158 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.545002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:35845 (local address 127.0.201.130:32939)
0925 22:46:13.545119 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.545124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.545136 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:46:13.545173 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.545176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.545218 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.545301 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:13.545582 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.546160 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.547191 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.547593 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.547744 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.547796 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.547983 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.548047 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:46:13.548174 8168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.544840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:32939 (local address 127.0.201.131:35845)
0925 22:46:13.545022 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.545035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.545100 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.545308 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.545311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.545326 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:13.545434 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.545448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.546269 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.546275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.547070 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.547078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.547739 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.547742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.547847 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.547869 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.547970 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.548042 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 22:46:13.548763 7975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eae66701894675bdd63455f04d3a86" candidate_uuid: "f4af0e5f397c45908604223c8fff2b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd22e79ff714a928cd1116f0261e49e" is_pre_election: true
I20260925 22:46:13.549055 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.544824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44531 (local address 127.0.201.131:49929)
0925 22:46:13.544954 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.544969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.545049 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.545511 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.545516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.545535 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:13.545638 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.545643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.546588 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.546591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.547785 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.547795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.548696 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.548700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.548792 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.548814 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.548878 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.548938 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1}
I20260925 22:46:13.549038 7975 raft_consensus.cc:2454] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4af0e5f397c45908604223c8fff2b4f in term 0.
I20260925 22:46:13.549463 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.545221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:49929 (local address 127.0.201.129:44531)
0925 22:46:13.545315 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.545319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.545329 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:46:13.545378 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.545380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.545416 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.545483 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:13.545722 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.546464 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.547914 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.548503 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.548604 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.548644 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.549279 (+ 635us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.549349 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:46:13.549474 8048 leader_election.cc:302] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cd22e79ff714a928cd1116f0261e49e, f4af0e5f397c45908604223c8fff2b4f; no voters:
I20260925 22:46:13.549911 7838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eae66701894675bdd63455f04d3a86" candidate_uuid: "f4af0e5f397c45908604223c8fff2b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427fbaeacb8b4c66ad3417551bc7174c" is_pre_election: true
I20260925 22:46:13.550154 7838 raft_consensus.cc:2454] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4af0e5f397c45908604223c8fff2b4f in term 0.
I20260925 22:46:13.550398 8201 raft_consensus.cc:2781] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:13.550490 8201 raft_consensus.cc:482] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:13.550549 8201 raft_consensus.cc:3037] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:13.551647 8201 raft_consensus.cc:504] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } } peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } }
I20260925 22:46:13.552042 8201 leader_election.cc:288] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [CANDIDATE]: Term 1 election: Requested vote from peers 7cd22e79ff714a928cd1116f0261e49e (127.0.201.130:32939), 427fbaeacb8b4c66ad3417551bc7174c (127.0.201.129:44531)
I20260925 22:46:13.552302 7975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eae66701894675bdd63455f04d3a86" candidate_uuid: "f4af0e5f397c45908604223c8fff2b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd22e79ff714a928cd1116f0261e49e"
I20260925 22:46:13.552322 7838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eae66701894675bdd63455f04d3a86" candidate_uuid: "f4af0e5f397c45908604223c8fff2b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427fbaeacb8b4c66ad3417551bc7174c"
I20260925 22:46:13.552426 7975 raft_consensus.cc:3037] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:13.552457 7838 raft_consensus.cc:3037] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:13.553372 7838 raft_consensus.cc:2454] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4af0e5f397c45908604223c8fff2b4f in term 1.
I20260925 22:46:13.553612 7975 raft_consensus.cc:2454] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4af0e5f397c45908604223c8fff2b4f in term 1.
I20260925 22:46:13.553642 8047 leader_election.cc:302] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427fbaeacb8b4c66ad3417551bc7174c, f4af0e5f397c45908604223c8fff2b4f; no voters:
I20260925 22:46:13.553795 8201 raft_consensus.cc:2781] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:13.554028 8201 raft_consensus.cc:686] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 1 LEADER]: Becoming Leader. State: Replica: f4af0e5f397c45908604223c8fff2b4f, State: Running, Role: LEADER
I20260925 22:46:13.554351 8201 consensus_queue.cc:237] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } } peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } }
I20260925 22:46:13.558007 6914 catalog_manager.cc:5495] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to f4af0e5f397c45908604223c8fff2b4f (127.0.201.131). New cstate: current_term: 1 leader_uuid: "f4af0e5f397c45908604223c8fff2b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } health_report { overall_health: HEALTHY } } }
I20260925 22:46:13.582912 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.579226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39204 (local address 127.0.201.131:36793)
0925 22:46:13.579371 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.579377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.579395 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:46:13.579487 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.579489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.579540 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.579652 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:13.579934 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.580536 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.581463 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.581682 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.581785 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.582116 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:13.582537 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:13.582672 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.582714 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.582768 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:46:13.588862 7838 raft_consensus.cc:1261] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c [term 1 FOLLOWER]: Refusing update from remote peer f4af0e5f397c45908604223c8fff2b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:13.588900 7975 raft_consensus.cc:1261] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e [term 1 FOLLOWER]: Refusing update from remote peer f4af0e5f397c45908604223c8fff2b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:13.589584 8202 consensus_queue.cc:1035] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:13.589769 8201 consensus_queue.cc:1035] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:13.597723 8205 mvcc.cc:205] Tried to move back new op lower bound from 7333381626211917824 to 7333381626081517568. Current Snapshot: MvccSnapshot[applied={T|T < 7333381626211917824}]
I20260925 22:46:13.600785 8206 mvcc.cc:205] Tried to move back new op lower bound from 7333381626211917824 to 7333381626081517568. Current Snapshot: MvccSnapshot[applied={T|T < 7333381626211917824}]
I20260925 22:46:13.601080 8207 mvcc.cc:205] Tried to move back new op lower bound from 7333381626211917824 to 7333381626081517568. Current Snapshot: MvccSnapshot[applied={T|T < 7333381626211917824}]
I20260925 22:46:13.608052 6915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:13.608218 6915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:13.612525 6915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16a9d45479b44fefbc4803169474a9a5: Sending DeleteTablet for 3 replicas of tablet a6eae66701894675bdd63455f04d3a86
I20260925 22:46:13.613480 7818 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eae66701894675bdd63455f04d3a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:13 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:55588
I20260925 22:46:13.613620 8093 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eae66701894675bdd63455f04d3a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:13 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:39192
I20260925 22:46:13.613874 7955 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eae66701894675bdd63455f04d3a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:13 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:34400
I20260925 22:46:13.614091 8215 tablet_replica.cc:330] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: stopping tablet replica
I20260925 22:46:13.614423 8215 raft_consensus.cc:2229] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:46:13.614800 8215 raft_consensus.cc:2258] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:13.615052 8216 tablet_replica.cc:330] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: stopping tablet replica
I20260925 22:46:13.615654 8216 raft_consensus.cc:2229] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:13.615998 8216 raft_consensus.cc:2258] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:13.616194 8215 ts_tablet_manager.cc:1905] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:13.616933 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 6883
I20260925 22:46:13.618983 8215 ts_tablet_manager.cc:1918] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:13.619069 8215 log.cc:1196] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/a6eae66701894675bdd63455f04d3a86
I20260925 22:46:13.619397 8215 ts_tablet_manager.cc:1939] T a6eae66701894675bdd63455f04d3a86 P f4af0e5f397c45908604223c8fff2b4f: Deleting consensus metadata
I20260925 22:46:13.624090 8214 tablet_replica.cc:330] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: stopping tablet replica
I20260925 22:46:13.624429 8214 raft_consensus.cc:2229] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:13.624745 8214 raft_consensus.cc:2258] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:46:13.625062 8048 connection.cc:535] server connection from 127.0.0.1:39192 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:46:13.625865 8216 ts_tablet_manager.cc:1905] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:13.626309 8214 ts_tablet_manager.cc:1905] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:13.628937 8214 ts_tablet_manager.cc:1918] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:13.629034 8214 log.cc:1196] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/a6eae66701894675bdd63455f04d3a86
I20260925 22:46:13.629169 8216 ts_tablet_manager.cc:1918] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:13.629307 8216 log.cc:1196] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/a6eae66701894675bdd63455f04d3a86
I20260925 22:46:13.629374 8214 ts_tablet_manager.cc:1939] T a6eae66701894675bdd63455f04d3a86 P 427fbaeacb8b4c66ad3417551bc7174c: Deleting consensus metadata
I20260925 22:46:13.629794 8216 ts_tablet_manager.cc:1939] T a6eae66701894675bdd63455f04d3a86 P 7cd22e79ff714a928cd1116f0261e49e: Deleting consensus metadata
W20260925 22:46:13.630643 7772 connection.cc:441] server connection from 127.0.0.1:55588 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55588 (request call id 1) could send its response
W20260925 22:46:13.630983 7911 connection.cc:441] server connection from 127.0.0.1:34400 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34400 (request call id 1) could send its response
I20260925 22:46:13.641566 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.615623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.0.201.189:35691)
0925 22:46:13.615867 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.615872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.616450 (+ 578us) server_negotiation.cc:407] Connection header received
0925 22:46:13.616643 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.616646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.616694 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.616749 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.618930 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.619988 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.621485 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.621935 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.622028 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.630538 (+ 8510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.630542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.630545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.630573 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.632521 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.632988 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.632992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.632994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.633081 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.633361 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.633364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.633366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.633581 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.633708 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.638650 (+ 4942us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.641395 (+ 2745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260925 22:46:13.644423 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.629346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.0.201.189:35691)
0925 22:46:13.629499 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.629505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.630204 (+ 699us) server_negotiation.cc:407] Connection header received
0925 22:46:13.630385 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.630388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.630438 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.630511 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.632197 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.633014 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.636352 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.636797 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.636916 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.638864 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.638869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.638872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.638896 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.641304 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.642951 (+ 1647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.642957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.642961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.643049 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.643377 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.643383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.643386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.643618 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.643767 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.644166 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.644272 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:46:13.645975 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.632128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.0.201.188:45327)
0925 22:46:13.632233 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.632238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.632916 (+ 678us) server_negotiation.cc:407] Connection header received
0925 22:46:13.633092 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.633095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.633143 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.633197 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.634014 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.634791 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.636030 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.636421 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.636511 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.638960 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.638964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.638967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.638992 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.643147 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.643817 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.643821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.643823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.643913 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.644191 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.644194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.644196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.644999 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.645127 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.645374 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.645784 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":1346304}
I20260925 22:46:13.649029 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.615748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.0.201.188:45327)
0925 22:46:13.615950 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.615955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.635037 (+ 19082us) server_negotiation.cc:407] Connection header received
0925 22:46:13.635309 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.635315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.635399 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.635513 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.636671 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.638350 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.640586 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.641122 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.641275 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.644331 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.644334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.644337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.644375 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.647434 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.647993 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.647995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.647997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.648085 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.648344 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.648346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.648347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.648505 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.648595 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.648806 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.648890 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260925 22:46:14.603356 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.602758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.131:34287)
0925 22:46:14.603125 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:14.603227 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":76,"threads_started":1}
I20260925 22:46:14.603485 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.603063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.129:54077)
0925 22:46:14.603272 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:14.603353 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
W20260925 22:46:14.603706 8160 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:33667 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
W20260925 22:46:14.603765 7885 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:33667 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
I20260925 22:46:14.604499 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.604134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.130:44699)
0925 22:46:14.604332 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:14.604424 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":52,"threads_started":1}
W20260925 22:46:14.604761 8022 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:33667 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
I20260925 22:46:15.112128 8228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16a9d45479b44fefbc4803169474a9a5)
I20260925 22:46:15.112305 8228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } }
I20260925 22:46:15.113232 8228 leader_election.cc:288] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667), 990efe788dd84cec88dad8e7cefbb68b (127.0.201.188:45327)
I20260925 22:46:15.113224 8229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16a9d45479b44fefbc4803169474a9a5)
I20260925 22:46:15.113412 8229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } }
I20260925 22:46:15.113902 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.113357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42704)
0925 22:46:15.113721 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.113796 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":83,"threads_started":1}
W20260925 22:46:15.114158 7499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
I20260925 22:46:15.114419 8229 leader_election.cc:288] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667), 6daee056e6dd48e2a9d24fd3451b9460 (127.0.201.189:35691)
I20260925 22:46:15.115053 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.114482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42708)
0925 22:46:15.114885 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.114953 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":65,"threads_started":1}
I20260925 22:46:15.115157 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.114904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42720)
0925 22:46:15.115041 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.115088 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 22:46:15.115195 7689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
W20260925 22:46:15.115294 7499 leader_election.cc:334] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667): Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
I20260925 22:46:15.116044 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.115813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42732)
0925 22:46:15.115922 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.115970 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:46:15.116145 7689 leader_election.cc:334] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667): Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
I20260925 22:46:15.118324 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.113943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.0.201.188:45327)
0925 22:46:15.114136 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:15.114141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:15.114156 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:46:15.114229 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:15.114232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:15.114280 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:15.114690 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:15.115048 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.115824 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.117326 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.117796 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.117924 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:15.117979 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:15.118137 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 22:46:15.118208 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
I20260925 22:46:15.118325 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.113781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:45327 (local address 127.0.0.1:52660)
0925 22:46:15.114009 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.114024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:15.114137 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:15.114775 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:15.114778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:15.114799 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:15.114915 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:15.114924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.115951 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.115954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:15.117216 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:15.117223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.117912 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.117915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:15.118007 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:15.118038 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 22:46:15.118123 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:46:15.118214 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260925 22:46:15.118781 7715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6daee056e6dd48e2a9d24fd3451b9460" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "990efe788dd84cec88dad8e7cefbb68b" is_pre_election: true
I20260925 22:46:15.118991 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.114655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.0.201.189:35691)
0925 22:46:15.115424 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:15.115436 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:15.115453 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:46:15.115580 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:15.115583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:15.115623 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:15.115722 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:15.116064 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.116786 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.117840 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.118272 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.118385 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:15.118472 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:15.118815 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 22:46:15.118883 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":665,"thread_start_us":78,"threads_started":1}
I20260925 22:46:15.118999 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.114604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:35691 (local address 127.0.0.1:41570)
0925 22:46:15.114974 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.114989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:15.115072 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:15.115819 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:15.115820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:15.115834 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:15.115946 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:15.115962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.116892 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.116894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:15.117716 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:15.117724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.118627 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.118629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:15.118692 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:15.118709 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:46:15.118801 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:46:15.118871 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":19,"thread_start_us":66,"threads_started":1}
I20260925 22:46:15.118974 7715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6daee056e6dd48e2a9d24fd3451b9460 in term 1.
I20260925 22:46:15.119417 7518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "990efe788dd84cec88dad8e7cefbb68b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6daee056e6dd48e2a9d24fd3451b9460" is_pre_election: true
I20260925 22:46:15.119429 7499 leader_election.cc:302] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6daee056e6dd48e2a9d24fd3451b9460, 990efe788dd84cec88dad8e7cefbb68b; no voters: 16a9d45479b44fefbc4803169474a9a5
I20260925 22:46:15.119606 7518 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 990efe788dd84cec88dad8e7cefbb68b in term 1.
I20260925 22:46:15.119720 8228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 22:46:15.119787 8228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 16a9d45479b44fefbc4803169474a9a5)
I20260925 22:46:15.119835 8228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:46:15.119935 7692 leader_election.cc:302] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6daee056e6dd48e2a9d24fd3451b9460, 990efe788dd84cec88dad8e7cefbb68b; no voters: 16a9d45479b44fefbc4803169474a9a5
I20260925 22:46:15.120138 8229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 22:46:15.120182 8229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 16a9d45479b44fefbc4803169474a9a5)
I20260925 22:46:15.120209 8229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:46:15.120910 8228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } }
I20260925 22:46:15.121227 8228 leader_election.cc:288] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [CANDIDATE]: Term 2 election: Requested vote from peers 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667), 990efe788dd84cec88dad8e7cefbb68b (127.0.201.188:45327)
I20260925 22:46:15.121167 8229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } }
I20260925 22:46:15.121572 8229 leader_election.cc:288] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 2 election: Requested vote from peers 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667), 6daee056e6dd48e2a9d24fd3451b9460 (127.0.201.189:35691)
I20260925 22:46:15.121593 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.121358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42734)
0925 22:46:15.121438 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.121525 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 22:46:15.121881 7715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6daee056e6dd48e2a9d24fd3451b9460" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "990efe788dd84cec88dad8e7cefbb68b"
I20260925 22:46:15.122061 7715 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6daee056e6dd48e2a9d24fd3451b9460 in current term 2: Already voted for candidate 990efe788dd84cec88dad8e7cefbb68b in this term.
I20260925 22:46:15.122184 7518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "990efe788dd84cec88dad8e7cefbb68b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6daee056e6dd48e2a9d24fd3451b9460"
I20260925 22:46:15.122279 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.122030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42738)
0925 22:46:15.122135 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.122188 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 22:46:15.122365 7518 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 990efe788dd84cec88dad8e7cefbb68b in current term 2: Already voted for candidate 6daee056e6dd48e2a9d24fd3451b9460 in this term.
W20260925 22:46:15.122462 7499 leader_election.cc:334] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667): Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
I20260925 22:46:15.122526 7499 leader_election.cc:302] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6daee056e6dd48e2a9d24fd3451b9460; no voters: 16a9d45479b44fefbc4803169474a9a5, 990efe788dd84cec88dad8e7cefbb68b
I20260925 22:46:15.122663 8228 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 22:46:15.122700 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.121630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42736)
0925 22:46:15.122591 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.122642 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":880}
I20260925 22:46:15.123373 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.123162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42742)
0925 22:46:15.123279 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.123330 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:46:15.123494 7689 leader_election.cc:334] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667): Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
I20260925 22:46:15.123582 7689 leader_election.cc:302] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 990efe788dd84cec88dad8e7cefbb68b; no voters: 16a9d45479b44fefbc4803169474a9a5, 6daee056e6dd48e2a9d24fd3451b9460
I20260925 22:46:15.123749 8229 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 22:46:15.604748 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.604278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.131:43299)
0925 22:46:15.604572 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.604641 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
I20260925 22:46:15.604748 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.604298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.129:49429)
0925 22:46:15.604608 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.604657 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260925 22:46:15.605751 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.605338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.130:59709)
0925 22:46:15.605606 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.605673 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":127,"threads_started":1}
I20260925 22:46:16.605876 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:16.605469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.129:50265)
0925 22:46:16.605736 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:16.605791 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260925 22:46:16.605883 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:16.605456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.131:34947)
0925 22:46:16.605731 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:16.605790 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":66,"threads_started":1}
I20260925 22:46:16.606673 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:16.606341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.130:40069)
0925 22:46:16.606554 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:16.606603 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260925 22:46:17.117937 8244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:17.118108 8244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } }
I20260925 22:46:17.118584 8244 leader_election.cc:288] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667), 6daee056e6dd48e2a9d24fd3451b9460 (127.0.201.189:35691)
I20260925 22:46:17.119067 7518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "990efe788dd84cec88dad8e7cefbb68b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6daee056e6dd48e2a9d24fd3451b9460" is_pre_election: true
I20260925 22:46:17.119134 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.118740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42972)
0925 22:46:17.118985 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.119053 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260925 22:46:17.119197 7518 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 990efe788dd84cec88dad8e7cefbb68b in term 2.
I20260925 22:46:17.119609 7692 leader_election.cc:302] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6daee056e6dd48e2a9d24fd3451b9460, 990efe788dd84cec88dad8e7cefbb68b; no voters:
I20260925 22:46:17.119824 8244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 22:46:17.119897 8244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:17.119927 8244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:46:17.120193 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.119981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42976)
0925 22:46:17.120095 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.120140 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 22:46:17.120293 7689 leader_election.cc:334] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667): Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
I20260925 22:46:17.121050 8244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } }
I20260925 22:46:17.121376 8244 leader_election.cc:288] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 3 election: Requested vote from peers 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667), 6daee056e6dd48e2a9d24fd3451b9460 (127.0.201.189:35691)
I20260925 22:46:17.121779 7518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "990efe788dd84cec88dad8e7cefbb68b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6daee056e6dd48e2a9d24fd3451b9460"
I20260925 22:46:17.121871 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.121493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42982)
0925 22:46:17.121745 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.121805 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260925 22:46:17.121902 7518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:46:17.122498 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.122300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:42992)
0925 22:46:17.122396 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.122445 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:46:17.122613 7689 leader_election.cc:334] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667): Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
I20260925 22:46:17.123035 7518 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 990efe788dd84cec88dad8e7cefbb68b in term 3.
I20260925 22:46:17.123309 7692 leader_election.cc:302] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6daee056e6dd48e2a9d24fd3451b9460, 990efe788dd84cec88dad8e7cefbb68b; no voters: 16a9d45479b44fefbc4803169474a9a5
I20260925 22:46:17.123476 8244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:46:17.123721 8244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [term 3 LEADER]: Becoming Leader. State: Replica: 990efe788dd84cec88dad8e7cefbb68b, State: Running, Role: LEADER
I20260925 22:46:17.123992 8244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } }
I20260925 22:46:17.125013 8247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 990efe788dd84cec88dad8e7cefbb68b. Latest consensus state: current_term: 3 leader_uuid: "990efe788dd84cec88dad8e7cefbb68b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:17.125128 8247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:17.125404 8249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:17.127655 8249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:17.128157 8249 catalog_manager.cc:1259] Loaded cluster ID: df6895d752294032b572668a61c7b7c7
I20260925 22:46:17.128201 8249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:17.128759 8249 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:17.129107 8249 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Loaded TSK: 0
I20260925 22:46:17.129752 8249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:46:17.194963 7704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:46:17.198158 7704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:17.201260 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.200977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:43008)
0925 22:46:17.201097 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.201159 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 22:46:17.201412 7518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [term 3 FOLLOWER]: Refusing update from remote peer 990efe788dd84cec88dad8e7cefbb68b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 22:46:17.202055 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.201799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.0.1:43024)
0925 22:46:17.201912 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.201971 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 22:46:17.201959 8244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 22:46:17.202171 7689 consensus_peers.cc:487] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b -> Peer 16a9d45479b44fefbc4803169474a9a5 (127.0.201.190:33667): Couldn't send request to peer 16a9d45479b44fefbc4803169474a9a5. Status: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 22:46:17.203467 8251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 990efe788dd84cec88dad8e7cefbb68b. Latest consensus state: current_term: 3 leader_uuid: "990efe788dd84cec88dad8e7cefbb68b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:17.203635 8251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:46:17.204795 8247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "990efe788dd84cec88dad8e7cefbb68b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:17.204979 8247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:17.205147 8252 mvcc.cc:205] Tried to move back new op lower bound from 7333381641011658752 to 7333381640702332928. Current Snapshot: MvccSnapshot[applied={T|T < 7333381641011658752}]
I20260925 22:46:17.205008 8251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "990efe788dd84cec88dad8e7cefbb68b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "16a9d45479b44fefbc4803169474a9a5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33667 } } peers { permanent_uuid: "6daee056e6dd48e2a9d24fd3451b9460" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 35691 } attrs { promote: false } } peers { permanent_uuid: "990efe788dd84cec88dad8e7cefbb68b" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 45327 } attrs { promote: false } } }
I20260925 22:46:17.205477 8251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6daee056e6dd48e2a9d24fd3451b9460 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:46:17.215564 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.210742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36793 (local address 127.0.0.1:34444)
0925 22:46:17.211187 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.211203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:17.211304 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:17.211566 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:17.211568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:17.211585 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:17.211808 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:17.211815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.212840 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.212843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:17.214068 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:17.214078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.215135 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.215139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:17.215230 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.215250 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:46:17.215313 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 22:46:17.215423 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":33}
I20260925 22:46:17.215643 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.210875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.0.201.131:36793)
0925 22:46:17.211149 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:17.211154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:17.211273 (+ 119us) server_negotiation.cc:407] Connection header received
0925 22:46:17.211416 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:17.211419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:17.211466 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:17.211586 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:17.211947 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.212704 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.214205 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.214916 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.215017 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.215078 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:17.215429 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 22:46:17.215515 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
I20260925 22:46:17.216051 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.211588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.0.201.129:44531)
0925 22:46:17.211842 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:17.211849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:17.211870 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:46:17.211992 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:17.211995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:17.212040 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:17.212118 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:17.212607 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.213274 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.214410 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.214854 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.214972 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.215045 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:17.215846 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 22:46:17.215917 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260925 22:46:17.216308 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.211231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44531 (local address 127.0.0.1:60838)
0925 22:46:17.211809 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.211822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:17.211900 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:17.212219 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:17.212222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:17.212237 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:17.212312 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:17.212317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.213381 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.213387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:17.214290 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:17.214297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.215100 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.215103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:17.215194 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.215228 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:46:17.215298 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:46:17.216185 (+ 887us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":5,"thread_start_us":92,"threads_started":1}
I20260925 22:46:17.216599 8093 tablet_service.cc:1467] Processing CreateTablet for tablet bc8b46a540ea4ccd9d20dbd88711cc06 (DEFAULT_TABLE table=test-table [id=2809695cdf8044308e8d8f28275e54e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:17.216858 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.212519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:32939 (local address 127.0.0.1:57540)
0925 22:46:17.213177 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.213189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:17.213277 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:17.213550 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:17.213552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:17.213570 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:17.213657 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:17.213667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.214607 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.214610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:17.215713 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:17.215724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.216470 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.216473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:17.216555 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.216576 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:46:17.216661 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:46:17.216733 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":84,"threads_started":1}
I20260925 22:46:17.216938 8093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8b46a540ea4ccd9d20dbd88711cc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:17.216908 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.212825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.0.201.130:32939)
0925 22:46:17.213062 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:17.213066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:17.213354 (+ 288us) server_negotiation.cc:407] Connection header received
0925 22:46:17.213403 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:17.213406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:17.213447 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:17.213525 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:17.213776 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.214475 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.215848 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.216368 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.216471 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.216527 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:17.216689 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 22:46:17.216763 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260925 22:46:17.217020 7818 tablet_service.cc:1467] Processing CreateTablet for tablet bc8b46a540ea4ccd9d20dbd88711cc06 (DEFAULT_TABLE table=test-table [id=2809695cdf8044308e8d8f28275e54e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:17.217320 7818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8b46a540ea4ccd9d20dbd88711cc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:17.217545 7955 tablet_service.cc:1467] Processing CreateTablet for tablet bc8b46a540ea4ccd9d20dbd88711cc06 (DEFAULT_TABLE table=test-table [id=2809695cdf8044308e8d8f28275e54e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:17.217758 7955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8b46a540ea4ccd9d20dbd88711cc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:17.219337 8262 tablet_bootstrap.cc:492] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c: Bootstrap starting.
I20260925 22:46:17.220285 8262 tablet_bootstrap.cc:654] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:17.220376 8263 tablet_bootstrap.cc:492] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f: Bootstrap starting.
I20260925 22:46:17.220515 8264 tablet_bootstrap.cc:492] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e: Bootstrap starting.
I20260925 22:46:17.221268 8264 tablet_bootstrap.cc:654] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:17.221390 8263 tablet_bootstrap.cc:654] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:17.221616 8262 tablet_bootstrap.cc:492] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c: No bootstrap required, opened a new log
I20260925 22:46:17.221712 8262 ts_tablet_manager.cc:1397] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:17.222100 8262 raft_consensus.cc:348] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } } peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } }
I20260925 22:46:17.222193 8262 raft_consensus.cc:374] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:17.222213 8262 raft_consensus.cc:729] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427fbaeacb8b4c66ad3417551bc7174c, State: Initialized, Role: FOLLOWER
I20260925 22:46:17.222354 8262 consensus_queue.cc:260] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } } peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } }
I20260925 22:46:17.222500 8263 tablet_bootstrap.cc:492] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f: No bootstrap required, opened a new log
I20260925 22:46:17.222677 8262 ts_tablet_manager.cc:1428] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:17.222617 8263 ts_tablet_manager.cc:1397] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:17.223637 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.223193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.129:59975)
0925 22:46:17.223434 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.223498 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260925 22:46:17.224256 8263 raft_consensus.cc:348] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } } peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } }
I20260925 22:46:17.224349 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.224101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.129:57685)
0925 22:46:17.224236 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.224289 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 22:46:17.224376 8263 raft_consensus.cc:374] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:17.224416 8263 raft_consensus.cc:729] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4af0e5f397c45908604223c8fff2b4f, State: Initialized, Role: FOLLOWER
I20260925 22:46:17.224643 8263 consensus_queue.cc:260] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } } peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } }
I20260925 22:46:17.225004 8263 ts_tablet_manager.cc:1428] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:17.225630 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.225260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.131:39231)
0925 22:46:17.225487 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.225541 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260925 22:46:17.225803 7884 heartbeater.cc:503] Master 127.0.201.188:45327 was elected leader, sending a full tablet report...
I20260925 22:46:17.226567 8159 heartbeater.cc:503] Master 127.0.201.188:45327 was elected leader, sending a full tablet report...
I20260925 22:46:17.226686 8264 tablet_bootstrap.cc:492] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e: No bootstrap required, opened a new log
I20260925 22:46:17.226769 8264 ts_tablet_manager.cc:1397] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 22:46:17.227201 8264 raft_consensus.cc:348] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } } peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } }
I20260925 22:46:17.227319 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.226644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.131:32893)
0925 22:46:17.227200 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.227252 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494}
I20260925 22:46:17.227396 8264 raft_consensus.cc:374] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:17.227455 8264 raft_consensus.cc:729] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd22e79ff714a928cd1116f0261e49e, State: Initialized, Role: FOLLOWER
I20260925 22:46:17.227727 8264 consensus_queue.cc:260] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } } peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } }
I20260925 22:46:17.228096 8264 ts_tablet_manager.cc:1428] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:17.229200 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.228353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.130:53745)
0925 22:46:17.229020 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.229084 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":165,"threads_started":1}
I20260925 22:46:17.229470 8021 heartbeater.cc:503] Master 127.0.201.188:45327 was elected leader, sending a full tablet report...
I20260925 22:46:17.230168 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.229658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.130:39769)
0925 22:46:17.229890 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.230073 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":90,"threads_started":1}
W20260925 22:46:17.272588 7887 tablet.cc:2369] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:17.311214 8272 raft_consensus.cc:482] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:17.311370 8272 raft_consensus.cc:504] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } } peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } }
I20260925 22:46:17.312515 8272 leader_election.cc:288] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4af0e5f397c45908604223c8fff2b4f (127.0.201.131:36793), 427fbaeacb8b4c66ad3417551bc7174c (127.0.201.129:44531)
I20260925 22:46:17.316138 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.312713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36793 (local address 127.0.201.130:44519)
0925 22:46:17.312855 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.312873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:17.312964 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:17.313253 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:17.313256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:17.313276 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:17.313395 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:17.313400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.314131 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.314136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:17.315002 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:17.315010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.315646 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.315650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:17.315813 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.315830 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:46:17.315918 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:46:17.315985 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":71}
I20260925 22:46:17.316138 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.312840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:44519 (local address 127.0.201.131:36793)
0925 22:46:17.312981 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:17.312991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:17.313010 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:46:17.313071 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:17.313074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:17.313123 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:17.313235 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:17.313514 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.314022 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.315128 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.315512 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.315646 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.315720 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:17.315942 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 22:46:17.316010 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:46:17.316167 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.313088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:44689 (local address 127.0.201.129:44531)
0925 22:46:17.313228 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:17.313232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:17.313248 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:46:17.313312 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:17.313317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:17.313359 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:17.313444 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:17.313707 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.314197 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.315178 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.315553 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.315683 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.315755 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:17.315972 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 22:46:17.316037 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:46:17.316166 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.312964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44531 (local address 127.0.201.130:44689)
0925 22:46:17.313124 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.313139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:17.313218 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:17.313453 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:17.313456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:17.313471 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:17.313588 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:17.313593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.314305 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.314310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:17.315055 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:17.315063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.315694 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.315698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:17.315854 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.315871 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:46:17.315957 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:46:17.316037 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":46}
I20260925 22:46:17.316886 7838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8b46a540ea4ccd9d20dbd88711cc06" candidate_uuid: "7cd22e79ff714a928cd1116f0261e49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427fbaeacb8b4c66ad3417551bc7174c" is_pre_election: true
I20260925 22:46:17.316905 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8b46a540ea4ccd9d20dbd88711cc06" candidate_uuid: "7cd22e79ff714a928cd1116f0261e49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4af0e5f397c45908604223c8fff2b4f" is_pre_election: true
I20260925 22:46:17.317050 7838 raft_consensus.cc:2454] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cd22e79ff714a928cd1116f0261e49e in term 0.
I20260925 22:46:17.317119 8113 raft_consensus.cc:2454] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cd22e79ff714a928cd1116f0261e49e in term 0.
I20260925 22:46:17.317435 7910 leader_election.cc:302] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427fbaeacb8b4c66ad3417551bc7174c, 7cd22e79ff714a928cd1116f0261e49e; no voters:
I20260925 22:46:17.317658 8272 raft_consensus.cc:2781] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:17.317715 8272 raft_consensus.cc:482] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:17.317759 8272 raft_consensus.cc:3037] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:17.318740 8272 raft_consensus.cc:504] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } } peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } }
I20260925 22:46:17.319059 8272 leader_election.cc:288] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [CANDIDATE]: Term 1 election: Requested vote from peers f4af0e5f397c45908604223c8fff2b4f (127.0.201.131:36793), 427fbaeacb8b4c66ad3417551bc7174c (127.0.201.129:44531)
I20260925 22:46:17.319358 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8b46a540ea4ccd9d20dbd88711cc06" candidate_uuid: "7cd22e79ff714a928cd1116f0261e49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4af0e5f397c45908604223c8fff2b4f"
I20260925 22:46:17.319459 7838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8b46a540ea4ccd9d20dbd88711cc06" candidate_uuid: "7cd22e79ff714a928cd1116f0261e49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427fbaeacb8b4c66ad3417551bc7174c"
I20260925 22:46:17.319504 8113 raft_consensus.cc:3037] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:17.319595 7838 raft_consensus.cc:3037] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:17.320488 7838 raft_consensus.cc:2454] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cd22e79ff714a928cd1116f0261e49e in term 1.
I20260925 22:46:17.320497 8113 raft_consensus.cc:2454] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cd22e79ff714a928cd1116f0261e49e in term 1.
I20260925 22:46:17.320745 7910 leader_election.cc:302] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427fbaeacb8b4c66ad3417551bc7174c, 7cd22e79ff714a928cd1116f0261e49e; no voters:
I20260925 22:46:17.320912 8272 raft_consensus.cc:2781] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:17.321117 8272 raft_consensus.cc:686] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 1 LEADER]: Becoming Leader. State: Replica: 7cd22e79ff714a928cd1116f0261e49e, State: Running, Role: LEADER
I20260925 22:46:17.321542 8272 consensus_queue.cc:237] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } } peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } }
I20260925 22:46:17.321779 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.321545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.130:56675)
0925 22:46:17.321649 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.321707 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 22:46:17.324241 7704 catalog_manager.cc:5495] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cd22e79ff714a928cd1116f0261e49e (127.0.201.130). New cstate: current_term: 1 leader_uuid: "7cd22e79ff714a928cd1116f0261e49e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cd22e79ff714a928cd1116f0261e49e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 } health_report { overall_health: UNKNOWN } } }
I20260925 22:46:17.341486 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.338230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.0.201.130:32939)
0925 22:46:17.338359 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:17.338363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:17.338376 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:46:17.338500 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:17.338502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:17.338555 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:17.338643 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:17.338847 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.339351 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.340191 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:17.340420 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:17.340505 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:17.340717 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:17.341137 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:17.341221 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:17.341270 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 22:46:17.341315 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:46:17.344133 7838 raft_consensus.cc:1261] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c [term 1 FOLLOWER]: Refusing update from remote peer 7cd22e79ff714a928cd1116f0261e49e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:17.344393 8113 raft_consensus.cc:1261] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f [term 1 FOLLOWER]: Refusing update from remote peer 7cd22e79ff714a928cd1116f0261e49e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:17.344662 8272 consensus_queue.cc:1035] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [LEADER]: Connected to new peer: Peer: permanent_uuid: "427fbaeacb8b4c66ad3417551bc7174c" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:17.344945 8275 consensus_queue.cc:1035] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4af0e5f397c45908604223c8fff2b4f" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:17.347055 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.346032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.129:34177)
0925 22:46:17.346900 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.346976 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":791}
I20260925 22:46:17.347553 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.347304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.131:46563)
0925 22:46:17.347419 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.347484 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 22:46:17.349318 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.349086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.130:52049)
0925 22:46:17.349197 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.349249 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 22:46:17.351154 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.350541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.131:48377)
0925 22:46:17.351035 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.351089 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428}
I20260925 22:46:17.352345 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.351494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.130:52427)
0925 22:46:17.352246 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.352295 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":690}
I20260925 22:46:17.356320 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.356007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33667 (local address 127.0.201.129:33707)
0925 22:46:17.356172 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.356249 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 22:46:17.358253 7704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:17.358402 7704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:17.362057 7704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 990efe788dd84cec88dad8e7cefbb68b: Sending DeleteTablet for 3 replicas of tablet bc8b46a540ea4ccd9d20dbd88711cc06
I20260925 22:46:17.362727 7955 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8b46a540ea4ccd9d20dbd88711cc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:17 UTC) from {username='oryx', principal='oryx/127.0.201.188@KRBTEST.COM'} at 127.0.0.1:57540
I20260925 22:46:17.362815 8093 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8b46a540ea4ccd9d20dbd88711cc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:17 UTC) from {username='oryx', principal='oryx/127.0.201.188@KRBTEST.COM'} at 127.0.0.1:34444
I20260925 22:46:17.363034 8288 tablet_replica.cc:330] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e: stopping tablet replica
I20260925 22:46:17.363121 7818 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8b46a540ea4ccd9d20dbd88711cc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:17 UTC) from {username='oryx', principal='oryx/127.0.201.188@KRBTEST.COM'} at 127.0.0.1:60838
I20260925 22:46:17.363152 8288 raft_consensus.cc:2229] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:46:17.363431 8290 tablet_replica.cc:330] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c: stopping tablet replica
I20260925 22:46:17.363590 8290 raft_consensus.cc:2229] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:17.363662 8288 raft_consensus.cc:2258] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:17.363778 8290 raft_consensus.cc:2258] T bc8b46a540ea4ccd9d20dbd88711cc06 P 427fbaeacb8b4c66ad3417551bc7174c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:17.364078 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 7659
I20260925 22:46:17.365322 8288 ts_tablet_manager.cc:1905] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:17.367259 8288 ts_tablet_manager.cc:1918] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:17.367349 8288 log.cc:1196] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/bc8b46a540ea4ccd9d20dbd88711cc06
I20260925 22:46:17.367600 8288 ts_tablet_manager.cc:1939] T bc8b46a540ea4ccd9d20dbd88711cc06 P 7cd22e79ff714a928cd1116f0261e49e: Deleting consensus metadata
I20260925 22:46:17.368409 7692 catalog_manager.cc:4841] TS 7cd22e79ff714a928cd1116f0261e49e (127.0.201.130:32939): tablet bc8b46a540ea4ccd9d20dbd88711cc06 (table test-table [id=2809695cdf8044308e8d8f28275e54e9]) successfully deleted
I20260925 22:46:17.372380 8289 tablet_replica.cc:330] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f: stopping tablet replica
I20260925 22:46:17.372470 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 7892
I20260925 22:46:17.372555 8289 raft_consensus.cc:2229] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:17.372604 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:17.372335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44531 (local address 127.0.0.1:60854)
0925 22:46:17.372483 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:17.372533 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:44531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 22:46:17.372696 8289 raft_consensus.cc:2258] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:46:17.373250 7691 catalog_manager.cc:4584] TS 427fbaeacb8b4c66ad3417551bc7174c (127.0.201.129:44531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc8b46a540ea4ccd9d20dbd88711cc06: Network error: Client connection negotiation failed: client connection to 127.0.201.129:44531: connect: Connection refused (error 111)
I20260925 22:46:17.373669 8289 ts_tablet_manager.cc:1905] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:17.375974 8289 ts_tablet_manager.cc:1918] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:17.376036 8289 log.cc:1196] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/bc8b46a540ea4ccd9d20dbd88711cc06
I20260925 22:46:17.376302 8289 ts_tablet_manager.cc:1939] T bc8b46a540ea4ccd9d20dbd88711cc06 P f4af0e5f397c45908604223c8fff2b4f: Deleting consensus metadata
I20260925 22:46:17.376897 7691 catalog_manager.cc:4841] TS f4af0e5f397c45908604223c8fff2b4f (127.0.201.131:36793): tablet bc8b46a540ea4ccd9d20dbd88711cc06 (table test-table [id=2809695cdf8044308e8d8f28275e54e9]) successfully deleted
I20260925 22:46:17.378580 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 8028
I20260925 22:46:17.385025 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 7379
I20260925 22:46:17.390656 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 7487
2026-09-25T22:46:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7289 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:46:17 dist-test-slave-vdtf krb5kdc[8293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:46:17 dist-test-slave-vdtf krb5kdc[8293](info): set up 2 sockets
Sep 25 22:46:17 dist-test-slave-vdtf krb5kdc[8293](info): commencing operation
krb5kdc: starting...
W20260925 22:46:18.448232 806 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:46:18 dist-test-slave-vdtf krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.190.keytab.
Entry for principal oryx/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.190.keytab.
I20260925 22:46:18.566605 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40195
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.190.keytab
--principal=oryx/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:40195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:18.677353 8306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:18.677590 8306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:18.677628 8306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:18.681511 8306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:18.681558 8306 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:46:18.681576 8306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:18.681593 8306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:18.681609 8306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:18.681624 8306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:18.686812 8306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:40195
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.190.keytab
--principal=oryx/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:18.687753 8306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:18.688663 8306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:18.696805 8311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:18.696841 8312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:18.696828 8314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:18.696945 8306 server_base.cc:1034] running on GCE node
I20260925 22:46:18.697696 8306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:18.698702 8306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:18.699858 8306 hybrid_clock.cc:648] HybridClock initialized: now 1790376378699840 us; error 29 us; skew 500 ppm
Sep 25 22:46:18 dist-test-slave-vdtf krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376378, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:18.702477 8306 init.cc:377] Logged in from keytab as oryx/127.0.201.190@KRBTEST.COM (short username oryx)
I20260925 22:46:18.703263 8306 webserver.cc:460] Webserver started at http://127.0.201.190:35711/ using document root <none> and password file <none>
I20260925 22:46:18.703825 8306 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:18.703890 8306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:18.704059 8306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:18.705952 8306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "ce5569581c1243b9afc87e91506d9103"
format_stamp: "Formatted at 2026-09-25 22:46:18 on dist-test-slave-vdtf"
server_key: "b5d0b05fbf6e22f05419bb33b7fcad6d"
server_key_iv: "6ce4a7647294c598dc800fc7ad9100ab"
server_key_version: "encryptionkey@0"
I20260925 22:46:18.706446 8306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "ce5569581c1243b9afc87e91506d9103"
format_stamp: "Formatted at 2026-09-25 22:46:18 on dist-test-slave-vdtf"
server_key: "b5d0b05fbf6e22f05419bb33b7fcad6d"
server_key_iv: "6ce4a7647294c598dc800fc7ad9100ab"
server_key_version: "encryptionkey@0"
I20260925 22:46:18.710216 8306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:18.712752 8321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:18.714054 8306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:18.714211 8306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "ce5569581c1243b9afc87e91506d9103"
format_stamp: "Formatted at 2026-09-25 22:46:18 on dist-test-slave-vdtf"
server_key: "b5d0b05fbf6e22f05419bb33b7fcad6d"
server_key_iv: "6ce4a7647294c598dc800fc7ad9100ab"
server_key_version: "encryptionkey@0"
I20260925 22:46:18.714330 8306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:18.726280 8306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:18.729081 8306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:18.729274 8306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:18.736554 8306 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:40195
I20260925 22:46:18.736579 8383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:40195 every 8 connection(s)
I20260925 22:46:18.737444 8306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:18.740031 8384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:18.742014 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 8306
I20260925 22:46:18.742164 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:18.742483 806 external_mini_cluster.cc:1428] Setting key 9ffa9a75954408da7e3391199dd68747
I20260925 22:46:18.745679 8384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103: Bootstrap starting.
I20260925 22:46:18.747844 8384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:18.748553 8384 log.cc:824] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:18.750252 8384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103: No bootstrap required, opened a new log
Sep 25 22:46:18 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:18.754743 8384 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5569581c1243b9afc87e91506d9103" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40195 } }
I20260925 22:46:18.755018 8384 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:18.755131 8384 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5569581c1243b9afc87e91506d9103, State: Initialized, Role: FOLLOWER
I20260925 22:46:18.755666 8384 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5569581c1243b9afc87e91506d9103" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40195 } }
I20260925 22:46:18.755867 8384 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:18.755936 8384 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:18.756034 8384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:18.756644 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:18.743910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.0.201.190:40195)
0925 22:46:18.744348 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:18.744362 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:18.744397 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:46:18.745013 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:18.745029 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:18.745340 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:18.745677 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:18.746457 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:18.747409 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:18.748358 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:18.748855 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:18.749040 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:18.751614 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:18.751620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:18.751635 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:18.751664 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:18.753554 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:18.753972 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:18.753978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:18.753984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:18.754124 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:18.754407 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:18.754411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:18.754412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:18.754882 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:18.755086 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:18.755397 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 22:46:18.755745 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":154,"threads_started":1}
I20260925 22:46:18.756969 8384 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5569581c1243b9afc87e91506d9103" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40195 } }
I20260925 22:46:18.757311 8384 leader_election.cc:302] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce5569581c1243b9afc87e91506d9103; no voters:
I20260925 22:46:18.757611 8384 leader_election.cc:288] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:18.757709 8389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:18.757905 8389 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [term 1 LEADER]: Becoming Leader. State: Replica: ce5569581c1243b9afc87e91506d9103, State: Running, Role: LEADER
I20260925 22:46:18.758188 8389 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5569581c1243b9afc87e91506d9103" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40195 } }
I20260925 22:46:18.759668 8390 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce5569581c1243b9afc87e91506d9103. Latest consensus state: current_term: 1 leader_uuid: "ce5569581c1243b9afc87e91506d9103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5569581c1243b9afc87e91506d9103" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40195 } } }
I20260925 22:46:18.759806 8390 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:18.759991 8384 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:18.760145 8393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:18.760195 8391 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce5569581c1243b9afc87e91506d9103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5569581c1243b9afc87e91506d9103" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40195 } } }
I20260925 22:46:18.760830 8391 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:18.763227 8393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:18.768420 8393 catalog_manager.cc:1347] Generated new cluster ID: db98a532e5794765bae5781c8f04c411
I20260925 22:46:18.768479 8393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:18.801738 8393 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:18.802995 8393 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:18.811460 8393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103: Generated new TSK 0
I20260925 22:46:18.812297 8393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab.
Entry for principal oryx/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab.
I20260925 22:46:18.850524 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40195
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab
--principal=oryx/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:18.959270 8410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:18.959549 8410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:18.959617 8410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:18.963999 8410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:18.964075 8410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:18.964143 8410 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:46:18.964224 8410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:18.969285 8410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab
--principal=oryx/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.201.190:40195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:18.970341 8410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:18.971400 8410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:18.980894 8415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:18.980964 8416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:18.981061 8410 server_base.cc:1034] running on GCE node
W20260925 22:46:18.981158 8418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:18.981498 8410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:18.982097 8410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:18.983281 8410 hybrid_clock.cc:648] HybridClock initialized: now 1790376378983269 us; error 25 us; skew 500 ppm
Sep 25 22:46:18 dist-test-slave-vdtf krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376378, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:18.985934 8410 init.cc:377] Logged in from keytab as oryx/127.0.201.129@KRBTEST.COM (short username oryx)
I20260925 22:46:18.986820 8410 webserver.cc:460] Webserver started at http://127.0.201.129:37891/ using document root <none> and password file <none>
I20260925 22:46:18.987360 8410 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:18.987448 8410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:18.987717 8410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:18.989679 8410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "6ae706988b66441abbb98f928d8b30ec"
format_stamp: "Formatted at 2026-09-25 22:46:18 on dist-test-slave-vdtf"
server_key: "08e6022d6c913dd5be7b22f90d01d1e8"
server_key_iv: "7d3e37fea4cbeef16f4d88b767dcce08"
server_key_version: "encryptionkey@0"
I20260925 22:46:18.990228 8410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "6ae706988b66441abbb98f928d8b30ec"
format_stamp: "Formatted at 2026-09-25 22:46:18 on dist-test-slave-vdtf"
server_key: "08e6022d6c913dd5be7b22f90d01d1e8"
server_key_iv: "7d3e37fea4cbeef16f4d88b767dcce08"
server_key_version: "encryptionkey@0"
I20260925 22:46:18.993755 8410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:18.996076 8425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:18.997225 8410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:18.997354 8410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "6ae706988b66441abbb98f928d8b30ec"
format_stamp: "Formatted at 2026-09-25 22:46:18 on dist-test-slave-vdtf"
server_key: "08e6022d6c913dd5be7b22f90d01d1e8"
server_key_iv: "7d3e37fea4cbeef16f4d88b767dcce08"
server_key_version: "encryptionkey@0"
I20260925 22:46:18.997458 8410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:19.016757 8410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:19.019857 8410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:19.020063 8410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:19.021425 8410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:19.021508 8410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:19.021592 8410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:19.021631 8410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:46:19 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376378, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:19.032236 8410 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:46151
I20260925 22:46:19.032299 8540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:46151 every 8 connection(s)
I20260925 22:46:19.033358 8410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:19.035794 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.022279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:48991 (local address 127.0.201.190:40195)
0925 22:46:19.022421 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.022426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.023278 (+ 852us) server_negotiation.cc:407] Connection header received
0925 22:46:19.024433 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.024439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.024505 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.024612 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:19.026079 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.026949 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.028681 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.029122 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.029237 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.032084 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:19.032088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:19.032091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:19.032121 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:19.033881 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.034484 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.034487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.034488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.034570 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.034824 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.034826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.034827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.034994 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:19.035068 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.035502 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.035607 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:46:19.036595 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.022697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40195 (local address 127.0.201.129:48991)
0925 22:46:19.023116 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.023162 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.024189 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.024810 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.024822 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.025246 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:19.025917 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.025939 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.027065 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.027068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.028536 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.028548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.029260 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.029262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.029391 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.029938 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:19.029953 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:19.031915 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:19.034031 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.034051 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.034070 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.034376 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.034634 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.034636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.034637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.034745 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.035091 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:19.035096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:19.035301 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.035443 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.035737 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
I20260925 22:46:19.036679 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 8410
I20260925 22:46:19.036808 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:19.037097 806 external_mini_cluster.cc:1428] Setting key 22cc280746bb17ff945108d3272bfbc2
I20260925 22:46:19.038074 8541 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40195
I20260925 22:46:19.038211 8541 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:19.038789 8541 heartbeater.cc:511] Master 127.0.201.190:40195 requested a full tablet report, sending...
I20260925 22:46:19.040469 8337 ts_manager.cc:194] Registered new tserver with Master: 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151)
I20260925 22:46:19.041925 8337 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:48991
WARNING: no policy specified for oryx/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:46:19.047777 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.040410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:58955 (local address 127.0.201.190:40195)
0925 22:46:19.040543 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.040547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.040762 (+ 215us) server_negotiation.cc:407] Connection header received
0925 22:46:19.040851 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.040856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.040908 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.040993 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:19.041465 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.042170 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.043074 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.043398 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.043501 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.044755 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:19.044759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:19.044762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:19.044784 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:19.046196 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.046609 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.046612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.046614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.046694 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.046963 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.046968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.046970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.047149 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:19.047255 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.047486 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.047596 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:46:19.047794 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.040354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40195 (local address 127.0.201.129:58955)
0925 22:46:19.040665 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.040680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.040775 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.041023 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.041026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.041227 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:19.041344 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.041351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.042293 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.042296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.042949 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.042957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.043505 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.043509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.043663 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.044244 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:19.044258 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:19.044617 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:19.046305 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.046309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.046311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.046503 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.046789 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.046792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.046794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.046877 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.047245 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:19.047248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:19.047337 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.047430 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.047573 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":4,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab.
Entry for principal oryx/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab.
I20260925 22:46:19.064909 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40195
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab
--principal=oryx/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:19.174028 8550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:19.174263 8550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:19.174305 8550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:19.178264 8550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:19.178313 8550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:19.178350 8550 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:46:19.178440 8550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:19.183502 8550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab
--principal=oryx/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.201.190:40195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:19.184571 8550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:19.185537 8550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:19.195098 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:19.195122 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:19.195346 8550 server_base.cc:1034] running on GCE node
W20260925 22:46:19.195113 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:19.195878 8550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:19.196494 8550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:19.197657 8550 hybrid_clock.cc:648] HybridClock initialized: now 1790376379197635 us; error 32 us; skew 500 ppm
Sep 25 22:46:19 dist-test-slave-vdtf krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376379, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:19.200613 8550 init.cc:377] Logged in from keytab as oryx/127.0.201.130@KRBTEST.COM (short username oryx)
I20260925 22:46:19.201498 8550 webserver.cc:460] Webserver started at http://127.0.201.130:37289/ using document root <none> and password file <none>
I20260925 22:46:19.202028 8550 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:19.202104 8550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:19.202344 8550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:19.204406 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "b89f55716f8847a3a8959047a7dd7a35"
format_stamp: "Formatted at 2026-09-25 22:46:19 on dist-test-slave-vdtf"
server_key: "01f62b0a2b77ddb15a545172aa64fbef"
server_key_iv: "a9406ced6bbbbbbc51084c279238f0af"
server_key_version: "encryptionkey@0"
I20260925 22:46:19.204937 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "b89f55716f8847a3a8959047a7dd7a35"
format_stamp: "Formatted at 2026-09-25 22:46:19 on dist-test-slave-vdtf"
server_key: "01f62b0a2b77ddb15a545172aa64fbef"
server_key_iv: "a9406ced6bbbbbbc51084c279238f0af"
server_key_version: "encryptionkey@0"
I20260925 22:46:19.208454 8550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:46:19.210785 8565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:19.212018 8550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:19.212158 8550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "b89f55716f8847a3a8959047a7dd7a35"
format_stamp: "Formatted at 2026-09-25 22:46:19 on dist-test-slave-vdtf"
server_key: "01f62b0a2b77ddb15a545172aa64fbef"
server_key_iv: "a9406ced6bbbbbbc51084c279238f0af"
server_key_version: "encryptionkey@0"
I20260925 22:46:19.212265 8550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:19.237816 8550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:19.241351 8550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:19.241561 8550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:19.242988 8550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:19.243072 8550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:19.243156 8550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:19.243207 8550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:46:19 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376379, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:19.253945 8550 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:38673
I20260925 22:46:19.254009 8680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:38673 every 8 connection(s)
I20260925 22:46:19.254956 8550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:19.257103 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.243845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:50115 (local address 127.0.201.190:40195)
0925 22:46:19.243990 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.243995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.244928 (+ 933us) server_negotiation.cc:407] Connection header received
0925 22:46:19.245981 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.245985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.246043 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.246099 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:19.247885 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.248808 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.250293 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.250520 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.250635 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.253459 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:19.253463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:19.253465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:19.253493 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:19.254938 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.255554 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.255560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.255564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.255712 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.256074 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.256078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.256080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.256297 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:19.256386 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.256866 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.256974 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:46:19.257814 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.244306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40195 (local address 127.0.201.130:50115)
0925 22:46:19.244754 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.244802 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.245736 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.246592 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.246601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.247022 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:19.247710 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.247733 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.248958 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.248961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.250157 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.250166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.250688 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.250692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.250890 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.251480 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:19.251495 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:19.253292 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:19.255053 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.255058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.255072 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.255421 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.255812 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.255815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.255817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.255955 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.256396 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:19.256400 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:19.256608 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.256753 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.256990 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":127,"threads_started":1}
I20260925 22:46:19.259087 8681 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40195
I20260925 22:46:19.259235 8681 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:19.260061 8681 heartbeater.cc:511] Master 127.0.201.190:40195 requested a full tablet report, sending...
I20260925 22:46:19.261123 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 8550
I20260925 22:46:19.261204 8337 ts_manager.cc:194] Registered new tserver with Master: b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130:38673)
I20260925 22:46:19.261252 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:19.261686 806 external_mini_cluster.cc:1428] Setting key 2bdc0120015df79b707e7b58804ed1c5
I20260925 22:46:19.261930 8337 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:50115
I20260925 22:46:19.269127 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.261348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40195 (local address 127.0.201.130:46407)
0925 22:46:19.261625 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.261650 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.261731 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.262425 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.262428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.262621 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:19.262754 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.262764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.263520 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.263523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.264228 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.264236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.264679 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.264683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.264773 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.265267 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:19.265287 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:19.265586 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:19.267580 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.267583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.267584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.267854 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.268154 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.268156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.268157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.268230 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.268732 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:19.268734 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:19.268803 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.268883 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.268990 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260925 22:46:19.269325 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.262107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:46407 (local address 127.0.201.190:40195)
0925 22:46:19.262220 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.262225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.262239 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:46:19.262278 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.262280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.262313 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.262407 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:19.262869 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.263395 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.264344 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.264554 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.264660 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.265726 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:19.265730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:19.265733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:19.265783 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:19.267470 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.267954 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.267961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.267965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.268054 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.268320 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.268325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.268330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.268634 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:19.268748 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.268971 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.269099 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for oryx/127.0.201.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab.
Entry for principal oryx/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab.
I20260925 22:46:19.290186 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40195
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab
--principal=oryx/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:19.398164 8690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:19.398397 8690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:19.398473 8690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:19.402621 8690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:19.402683 8690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:19.402720 8690 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:46:19.402815 8690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:19.407946 8690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab
--principal=oryx/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.201.190:40195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:19.408995 8690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:19.410022 8690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:19.419788 8696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:19.419781 8695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:19.419783 8698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:19.419869 8690 server_base.cc:1034] running on GCE node
I20260925 22:46:19.420437 8690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:19.421118 8690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:19.422281 8690 hybrid_clock.cc:648] HybridClock initialized: now 1790376379422245 us; error 50 us; skew 500 ppm
Sep 25 22:46:19 dist-test-slave-vdtf krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376379, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:19.425035 8690 init.cc:377] Logged in from keytab as oryx/127.0.201.131@KRBTEST.COM (short username oryx)
I20260925 22:46:19.425971 8690 webserver.cc:460] Webserver started at http://127.0.201.131:41333/ using document root <none> and password file <none>
I20260925 22:46:19.426524 8690 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:19.426586 8690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:19.426790 8690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:19.428877 8690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "c0f0bf025fef469a829de5257cd8c0cf"
format_stamp: "Formatted at 2026-09-25 22:46:19 on dist-test-slave-vdtf"
server_key: "f0c10e5c80d0eb7e7b11080fd1376efa"
server_key_iv: "fe23da8a8b4d1c51c88a4805bc4955b5"
server_key_version: "encryptionkey@0"
I20260925 22:46:19.429438 8690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "c0f0bf025fef469a829de5257cd8c0cf"
format_stamp: "Formatted at 2026-09-25 22:46:19 on dist-test-slave-vdtf"
server_key: "f0c10e5c80d0eb7e7b11080fd1376efa"
server_key_iv: "fe23da8a8b4d1c51c88a4805bc4955b5"
server_key_version: "encryptionkey@0"
I20260925 22:46:19.433020 8690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:46:19.435227 8705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:19.436439 8690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:19.436553 8690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "c0f0bf025fef469a829de5257cd8c0cf"
format_stamp: "Formatted at 2026-09-25 22:46:19 on dist-test-slave-vdtf"
server_key: "f0c10e5c80d0eb7e7b11080fd1376efa"
server_key_iv: "fe23da8a8b4d1c51c88a4805bc4955b5"
server_key_version: "encryptionkey@0"
I20260925 22:46:19.436648 8690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:19.449486 8690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:19.452243 8690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:19.452435 8690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:19.453846 8690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:19.453896 8690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:19.453934 8690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:19.453974 8690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:46:19 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376379, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:19.465180 8690 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:36649
I20260925 22:46:19.465205 8820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:36649 every 8 connection(s)
I20260925 22:46:19.466070 8690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:19.467464 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.455186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:46567 (local address 127.0.201.190:40195)
0925 22:46:19.455343 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.455349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.455800 (+ 451us) server_negotiation.cc:407] Connection header received
0925 22:46:19.456747 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.456750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.456806 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.456887 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:19.458061 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.458777 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.460134 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.460530 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.460652 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.463963 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:19.463966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:19.463969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:19.463999 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:19.465450 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.465924 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.465928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.465929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.466010 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.466361 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.466364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.466366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.466586 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:19.466675 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.467252 (+ 577us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.467351 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:46:19.468165 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.455149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40195 (local address 127.0.201.131:46567)
0925 22:46:19.455624 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.455696 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.456538 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.457164 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.457171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.457488 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:19.457918 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.457930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.458912 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.458920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.460007 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.460015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.460686 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.460690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.460911 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.461580 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:19.461604 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:19.463791 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:19.465558 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.465564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.465578 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.465820 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.466096 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.466100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.466102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.466263 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.466724 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:19.466729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:19.466999 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.467155 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.467374 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":140,"threads_started":1}
I20260925 22:46:19.469201 8821 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40195
I20260925 22:46:19.469358 8821 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:19.469808 8821 heartbeater.cc:511] Master 127.0.201.190:40195 requested a full tablet report, sending...
I20260925 22:46:19.470829 8338 ts_manager.cc:194] Registered new tserver with Master: c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:19.471570 8338 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:46567
I20260925 22:46:19.476126 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 8690
I20260925 22:46:19.476255 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:46:19.476601 806 external_mini_cluster.cc:1428] Setting key daeb2476aafac154513b2225fb1d44d0
I20260925 22:46:19.478041 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.471269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40195 (local address 127.0.201.131:52983)
0925 22:46:19.471548 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.471563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.471660 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.472132 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.472134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.472297 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:19.472389 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.472393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.473121 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.473125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.473665 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.473672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.474111 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.474115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.474205 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.474627 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:19.474637 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:19.474910 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:19.476508 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.476514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.476516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.476726 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.476979 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.476983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.476985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.477068 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.477533 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:19.477536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:19.477630 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.477735 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.477861 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260925 22:46:19.478108 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.471383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:52983 (local address 127.0.201.190:40195)
0925 22:46:19.471500 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.471504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.471635 (+ 131us) server_negotiation.cc:407] Connection header received
0925 22:46:19.471772 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.471774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.471835 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.471911 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:19.472523 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.473006 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.473775 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.474008 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.474112 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.475044 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:19.475048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:19.475050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:19.475072 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:19.476412 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.476805 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.476810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.476812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.476891 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.477181 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.477185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.477186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.477428 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:19.477523 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.477840 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.477975 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:46:19.478756 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:46:19 dist-test-slave-vdtf krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:46:19 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:19 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:46:19.501303 8387 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:19.495527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.0.201.190:40195)
0925 22:46:19.495652 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.495655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.495666 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:46:19.495743 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.495746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.495792 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.495857 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:19.496526 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.497022 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.497790 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.497992 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.498064 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.501163 (+ 3099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57300: BlockingRecv error: recv got EOF from 127.0.0.1:57300 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 22:46:19 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:19.510187 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.502586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.0.201.190:40195)
0925 22:46:19.502710 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.502713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.502759 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:46:19.502848 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.502850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.502883 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.502945 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:19.503464 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.504022 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.504806 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.504997 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.505068 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.507143 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:19.507146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:19.507148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:19.507167 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:19.508604 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.509018 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.509023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.509025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.509111 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.509387 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.509390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.509392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.509558 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:19.509691 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.509920 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.510033 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:46:19.512905 8338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:46:19.516011 8338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:19.529858 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.523951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:46151 (local address 127.0.0.1:38918)
0925 22:46:19.524723 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.524769 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.524889 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.526146 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.526148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.526165 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:19.526247 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.526252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.527239 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.527245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.528369 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.528378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.529374 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.529377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.529473 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.529510 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.529604 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.529685 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":15,"thread_start_us":78,"threads_started":1}
I20260925 22:46:19.529906 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.524133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.0.201.129:46151)
0925 22:46:19.525734 (+ 1601us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.525751 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.525772 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:46:19.525840 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.525845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.526029 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.526171 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:19.526357 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.527071 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.528500 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.529244 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.529353 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.529524 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.529655 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.529759 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1504,"thread_start_us":78,"threads_started":1}
I20260925 22:46:19.530233 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.524159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36649 (local address 127.0.0.1:53252)
0925 22:46:19.525108 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.525121 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.525206 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.525729 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.525735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.525755 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:19.525889 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.525897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.527368 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.527371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.528714 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.528724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.529780 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.529784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.529861 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.529891 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.529970 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.530116 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":34,"thread_start_us":82,"threads_started":1}
I20260925 22:46:19.531140 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.524417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:38673 (local address 127.0.0.1:57380)
0925 22:46:19.525405 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.525417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.525496 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.525935 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.525938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.525955 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:19.526038 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.526054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.527539 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.527543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.528714 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.528724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.530779 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.530783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.530866 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.530888 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.530947 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.531011 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260925 22:46:19.531226 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.524801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.0.201.130:38673)
0925 22:46:19.525255 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.525264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.525494 (+ 230us) server_negotiation.cc:407] Connection header received
0925 22:46:19.525584 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.525588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.525782 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.525935 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:19.526381 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.527430 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.529947 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.530657 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.530780 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.530936 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.530987 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.531074 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":102,"threads_started":1}
I20260925 22:46:19.532068 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.524714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.0.201.131:36649)
0925 22:46:19.525123 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.525139 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.525189 (+ 50us) server_negotiation.cc:407] Connection header received
0925 22:46:19.525349 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.525355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.525573 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.525723 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:19.526018 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.527247 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.528835 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.529643 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.529747 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.529886 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.531824 (+ 1938us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.531925 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":99,"threads_started":1}
I20260925 22:46:19.533322 8475 tablet_service.cc:1467] Processing CreateTablet for tablet e8cb78232b634f5ab3b48052d97f20fb (DEFAULT_TABLE table=test-table [id=50d7debf280b4487aaa26c41b885a79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:19.533812 8615 tablet_service.cc:1467] Processing CreateTablet for tablet e8cb78232b634f5ab3b48052d97f20fb (DEFAULT_TABLE table=test-table [id=50d7debf280b4487aaa26c41b885a79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:19.534111 8755 tablet_service.cc:1467] Processing CreateTablet for tablet e8cb78232b634f5ab3b48052d97f20fb (DEFAULT_TABLE table=test-table [id=50d7debf280b4487aaa26c41b885a79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:19.534543 8475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cb78232b634f5ab3b48052d97f20fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:19.534756 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cb78232b634f5ab3b48052d97f20fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:19.535044 8755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cb78232b634f5ab3b48052d97f20fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:19.541455 8849 tablet_bootstrap.cc:492] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: Bootstrap starting.
I20260925 22:46:19.542878 8850 tablet_bootstrap.cc:492] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: Bootstrap starting.
I20260925 22:46:19.543656 8849 tablet_bootstrap.cc:654] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:19.543991 8851 tablet_bootstrap.cc:492] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: Bootstrap starting.
I20260925 22:46:19.544651 8849 log.cc:824] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:19.545408 8850 tablet_bootstrap.cc:654] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:19.545846 8851 tablet_bootstrap.cc:654] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:19.546133 8850 log.cc:824] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:19.546599 8851 log.cc:824] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:19.546979 8849 tablet_bootstrap.cc:492] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: No bootstrap required, opened a new log
I20260925 22:46:19.547200 8849 ts_tablet_manager.cc:1397] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260925 22:46:19.547883 8850 tablet_bootstrap.cc:492] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: No bootstrap required, opened a new log
I20260925 22:46:19.548177 8850 ts_tablet_manager.cc:1397] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260925 22:46:19.548481 8851 tablet_bootstrap.cc:492] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: No bootstrap required, opened a new log
I20260925 22:46:19.548666 8851 ts_tablet_manager.cc:1397] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:46:19.550825 8850 raft_consensus.cc:348] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } }
I20260925 22:46:19.551059 8850 raft_consensus.cc:374] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:19.551131 8850 raft_consensus.cc:729] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae706988b66441abbb98f928d8b30ec, State: Initialized, Role: FOLLOWER
I20260925 22:46:19.551569 8850 consensus_queue.cc:260] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } }
I20260925 22:46:19.551741 8849 raft_consensus.cc:348] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } }
I20260925 22:46:19.552032 8849 raft_consensus.cc:374] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:19.552101 8849 raft_consensus.cc:729] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f0bf025fef469a829de5257cd8c0cf, State: Initialized, Role: FOLLOWER
I20260925 22:46:19.552332 8851 raft_consensus.cc:348] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } }
I20260925 22:46:19.552556 8851 raft_consensus.cc:374] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:19.552512 8850 ts_tablet_manager.cc:1428] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260925 22:46:19.552525 8849 consensus_queue.cc:260] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } }
I20260925 22:46:19.552665 8851 raft_consensus.cc:729] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89f55716f8847a3a8959047a7dd7a35, State: Initialized, Role: FOLLOWER
I20260925 22:46:19.552889 8541 heartbeater.cc:503] Master 127.0.201.190:40195 was elected leader, sending a full tablet report...
I20260925 22:46:19.553098 8851 consensus_queue.cc:260] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } }
I20260925 22:46:19.553498 8849 ts_tablet_manager.cc:1428] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260925 22:46:19.553730 8681 heartbeater.cc:503] Master 127.0.201.190:40195 was elected leader, sending a full tablet report...
I20260925 22:46:19.553782 8821 heartbeater.cc:503] Master 127.0.201.190:40195 was elected leader, sending a full tablet report...
I20260925 22:46:19.553970 8851 ts_tablet_manager.cc:1428] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:46:19.671192 8857 raft_consensus.cc:482] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:19.671461 8857 raft_consensus.cc:504] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } }
I20260925 22:46:19.672771 8857 leader_election.cc:288] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151), c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:19.676644 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.673018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:46151 (local address 127.0.201.130:47913)
0925 22:46:19.673114 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.673132 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.673217 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.673452 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.673455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.673473 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:19.673594 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.673599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.674525 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.674529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.675416 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.675425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.676294 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.676297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.676383 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.676402 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.676461 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.676543 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 22:46:19.676699 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.673088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:47913 (local address 127.0.201.129:46151)
0925 22:46:19.673207 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.673213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.673230 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:46:19.673293 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.673295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.673334 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.673438 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:19.673714 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.674391 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.675551 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.676179 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.676297 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.676362 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.676501 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.676570 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:46:19.677273 8492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cb78232b634f5ab3b48052d97f20fb" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae706988b66441abbb98f928d8b30ec" is_pre_election: true
I20260925 22:46:19.677593 8492 raft_consensus.cc:2454] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 0.
I20260925 22:46:19.677816 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.673100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:38667 (local address 127.0.201.131:36649)
0925 22:46:19.673214 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.673219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.673719 (+ 500us) server_negotiation.cc:407] Connection header received
0925 22:46:19.673838 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.673847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.673887 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.673964 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:19.674785 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.675310 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.676758 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.677275 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.677414 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.677459 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.677647 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.677716 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:46:19.677829 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.673020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36649 (local address 127.0.201.130:38667)
0925 22:46:19.673617 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.673628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.673748 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.674004 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.674007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.674023 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:19.674640 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.674654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.675423 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.675425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.676629 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.676638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.677414 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.677417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.677521 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.677537 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.677630 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.677692 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":251,"thread_start_us":157,"threads_started":1}
I20260925 22:46:19.678124 8569 leader_election.cc:302] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae706988b66441abbb98f928d8b30ec, b89f55716f8847a3a8959047a7dd7a35; no voters:
I20260925 22:46:19.678401 8857 raft_consensus.cc:2781] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:19.678481 8857 raft_consensus.cc:482] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:19.678519 8857 raft_consensus.cc:3037] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:19.678467 8775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cb78232b634f5ab3b48052d97f20fb" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f0bf025fef469a829de5257cd8c0cf" is_pre_election: true
I20260925 22:46:19.678737 8775 raft_consensus.cc:2454] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 0.
I20260925 22:46:19.679569 8857 raft_consensus.cc:504] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } }
I20260925 22:46:19.679949 8857 leader_election.cc:288] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 election: Requested vote from peers 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151), c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:19.680236 8492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cb78232b634f5ab3b48052d97f20fb" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae706988b66441abbb98f928d8b30ec"
I20260925 22:46:19.680290 8775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cb78232b634f5ab3b48052d97f20fb" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f0bf025fef469a829de5257cd8c0cf"
I20260925 22:46:19.680371 8775 raft_consensus.cc:3037] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:19.680369 8492 raft_consensus.cc:3037] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:19.681303 8775 raft_consensus.cc:2454] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 1.
I20260925 22:46:19.681442 8492 raft_consensus.cc:2454] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 1.
I20260925 22:46:19.681581 8566 leader_election.cc:302] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b89f55716f8847a3a8959047a7dd7a35, c0f0bf025fef469a829de5257cd8c0cf; no voters:
I20260925 22:46:19.681744 8857 raft_consensus.cc:2781] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:19.681978 8857 raft_consensus.cc:686] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 1 LEADER]: Becoming Leader. State: Replica: b89f55716f8847a3a8959047a7dd7a35, State: Running, Role: LEADER
I20260925 22:46:19.682294 8857 consensus_queue.cc:237] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } }
I20260925 22:46:19.685357 8338 catalog_manager.cc:5495] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130). New cstate: current_term: 1 leader_uuid: "b89f55716f8847a3a8959047a7dd7a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } health_report { overall_health: HEALTHY } } }
W20260925 22:46:19.718066 8822 tablet.cc:2369] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:19.719424 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.715108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.0.201.130:38673)
0925 22:46:19.715211 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.715217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.715237 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:46:19.715321 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.715324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.715387 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.715521 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:19.715797 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.716669 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.717551 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.717876 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.717995 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.718382 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:19.718930 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:19.719077 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.719173 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.719268 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 22:46:19.727104 8775 raft_consensus.cc:1261] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Refusing update from remote peer b89f55716f8847a3a8959047a7dd7a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:19.727104 8492 raft_consensus.cc:1261] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Refusing update from remote peer b89f55716f8847a3a8959047a7dd7a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:19.727759 8857 consensus_queue.cc:1035] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:19.727957 8859 consensus_queue.cc:1035] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:19.735901 8862 mvcc.cc:205] Tried to move back new op lower bound from 7333381651353858048 to 7333381651181002752. Current Snapshot: MvccSnapshot[applied={T|T < 7333381651353858048}]
I20260925 22:46:19.737262 8863 mvcc.cc:205] Tried to move back new op lower bound from 7333381651353858048 to 7333381651181002752. Current Snapshot: MvccSnapshot[applied={T|T < 7333381651353858048}]
I20260925 22:46:19.748571 8337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:19.748733 8337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:19.751235 8337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103: Sending DeleteTablet for 3 replicas of tablet e8cb78232b634f5ab3b48052d97f20fb
I20260925 22:46:19.752144 8615 tablet_service.cc:1514] Processing DeleteTablet for tablet e8cb78232b634f5ab3b48052d97f20fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:19 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:57380
I20260925 22:46:19.752149 8475 tablet_service.cc:1514] Processing DeleteTablet for tablet e8cb78232b634f5ab3b48052d97f20fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:19 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:38918
I20260925 22:46:19.752767 8755 tablet_service.cc:1514] Processing DeleteTablet for tablet e8cb78232b634f5ab3b48052d97f20fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:19 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:53252
I20260925 22:46:19.753202 8874 tablet_replica.cc:330] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: stopping tablet replica
I20260925 22:46:19.753515 8874 raft_consensus.cc:2229] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:19.753813 8874 raft_consensus.cc:2258] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:19.754210 8872 tablet_replica.cc:330] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: stopping tablet replica
I20260925 22:46:19.754175 8337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:46:19.754601 8337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:19.754781 8872 raft_consensus.cc:2229] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:19.755117 8872 raft_consensus.cc:2258] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:19.755520 8873 tablet_replica.cc:330] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: stopping tablet replica
I20260925 22:46:19.755839 8873 raft_consensus.cc:2229] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:46:19.757735 8874 ts_tablet_manager.cc:1905] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:19.758208 8872 ts_tablet_manager.cc:1905] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:19.758435 8873 raft_consensus.cc:2258] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:19.760466 8755 tablet_service.cc:1467] Processing CreateTablet for tablet 358ac482e1b241cc976fb326fb04ec0c (DEFAULT_TABLE table=test-table [id=1e995b210b954cdebd0c0c06b96d2354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:19.760660 8615 tablet_service.cc:1467] Processing CreateTablet for tablet 358ac482e1b241cc976fb326fb04ec0c (DEFAULT_TABLE table=test-table [id=1e995b210b954cdebd0c0c06b96d2354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:19.760905 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358ac482e1b241cc976fb326fb04ec0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:19.760903 8475 tablet_service.cc:1467] Processing CreateTablet for tablet 358ac482e1b241cc976fb326fb04ec0c (DEFAULT_TABLE table=test-table [id=1e995b210b954cdebd0c0c06b96d2354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:19.761121 8475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358ac482e1b241cc976fb326fb04ec0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:19.761312 8872 ts_tablet_manager.cc:1918] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:19.761396 8872 log.cc:1196] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/e8cb78232b634f5ab3b48052d97f20fb
I20260925 22:46:19.761706 8872 ts_tablet_manager.cc:1939] T e8cb78232b634f5ab3b48052d97f20fb P 6ae706988b66441abbb98f928d8b30ec: Deleting consensus metadata
I20260925 22:46:19.762918 8325 catalog_manager.cc:4841] TS 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151): tablet e8cb78232b634f5ab3b48052d97f20fb (table test-table [id=50d7debf280b4487aaa26c41b885a79a]) successfully deleted
I20260925 22:46:19.763269 8850 tablet_bootstrap.cc:492] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: Bootstrap starting.
I20260925 22:46:19.763363 8851 tablet_bootstrap.cc:492] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35: Bootstrap starting.
I20260925 22:46:19.764180 8850 tablet_bootstrap.cc:654] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:19.764731 8851 tablet_bootstrap.cc:654] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:19.765137 8873 ts_tablet_manager.cc:1905] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:19.766342 8755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358ac482e1b241cc976fb326fb04ec0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:19.768571 8874 ts_tablet_manager.cc:1918] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:19.768698 8874 log.cc:1196] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/e8cb78232b634f5ab3b48052d97f20fb
I20260925 22:46:19.768841 8849 tablet_bootstrap.cc:492] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf: Bootstrap starting.
I20260925 22:46:19.768988 8874 ts_tablet_manager.cc:1939] T e8cb78232b634f5ab3b48052d97f20fb P c0f0bf025fef469a829de5257cd8c0cf: Deleting consensus metadata
I20260925 22:46:19.769775 8849 tablet_bootstrap.cc:654] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:19.770730 8322 catalog_manager.cc:4841] TS c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649): tablet e8cb78232b634f5ab3b48052d97f20fb (table test-table [id=50d7debf280b4487aaa26c41b885a79a]) successfully deleted
I20260925 22:46:19.771242 8849 tablet_bootstrap.cc:492] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf: No bootstrap required, opened a new log
I20260925 22:46:19.771322 8849 ts_tablet_manager.cc:1397] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 22:46:19.771751 8849 raft_consensus.cc:348] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.771859 8849 raft_consensus.cc:374] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:19.771889 8849 raft_consensus.cc:729] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f0bf025fef469a829de5257cd8c0cf, State: Initialized, Role: FOLLOWER
I20260925 22:46:19.771996 8849 consensus_queue.cc:260] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.772236 8849 ts_tablet_manager.cc:1428] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:19.778306 8873 ts_tablet_manager.cc:1918] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:19.778390 8873 log.cc:1196] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/e8cb78232b634f5ab3b48052d97f20fb
I20260925 22:46:19.778693 8873 ts_tablet_manager.cc:1939] T e8cb78232b634f5ab3b48052d97f20fb P b89f55716f8847a3a8959047a7dd7a35: Deleting consensus metadata
I20260925 22:46:19.780041 8850 tablet_bootstrap.cc:492] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: No bootstrap required, opened a new log
I20260925 22:46:19.780130 8850 ts_tablet_manager.cc:1397] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260925 22:46:19.780468 8324 catalog_manager.cc:4841] TS b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130:38673): tablet e8cb78232b634f5ab3b48052d97f20fb (table test-table [id=50d7debf280b4487aaa26c41b885a79a]) successfully deleted
I20260925 22:46:19.780741 8850 raft_consensus.cc:348] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.780848 8850 raft_consensus.cc:374] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:19.780874 8850 raft_consensus.cc:729] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae706988b66441abbb98f928d8b30ec, State: Initialized, Role: FOLLOWER
I20260925 22:46:19.780989 8850 consensus_queue.cc:260] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.781230 8850 ts_tablet_manager.cc:1428] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:46:19.782172 8851 tablet_bootstrap.cc:492] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35: No bootstrap required, opened a new log
I20260925 22:46:19.782275 8851 ts_tablet_manager.cc:1397] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20260925 22:46:19.782821 8851 raft_consensus.cc:348] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.782960 8851 raft_consensus.cc:374] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:19.783000 8851 raft_consensus.cc:729] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89f55716f8847a3a8959047a7dd7a35, State: Initialized, Role: FOLLOWER
I20260925 22:46:19.783130 8851 consensus_queue.cc:260] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.783432 8851 ts_tablet_manager.cc:1428] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260925 22:46:19.784631 8542 tablet.cc:2369] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:19.874764 8855 raft_consensus.cc:482] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:19.874944 8855 raft_consensus.cc:504] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.876072 8855 leader_election.cc:288] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130:38673), c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:19.880730 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.876428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:52841 (local address 127.0.201.131:36649)
0925 22:46:19.876685 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.876689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.876765 (+ 76us) server_negotiation.cc:407] Connection header received
0925 22:46:19.876980 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.876986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.877048 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.877089 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:19.877600 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.878344 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.879462 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.880101 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.880229 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.880285 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.880528 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.880600 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260925 22:46:19.880759 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.876341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36649 (local address 127.0.201.129:52841)
0925 22:46:19.876672 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.876687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.876845 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.877263 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.877267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.877285 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:19.877467 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.877474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.878457 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.878459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.879261 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.879269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.880247 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.880252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.880374 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.880409 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.880508 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.880583 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":25,"spinlock_wait_cycles":19328,"thread_start_us":61,"threads_started":1}
I20260925 22:46:19.881306 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358ac482e1b241cc976fb326fb04ec0c" candidate_uuid: "6ae706988b66441abbb98f928d8b30ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f0bf025fef469a829de5257cd8c0cf" is_pre_election: true
I20260925 22:46:19.881472 8774 raft_consensus.cc:2454] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae706988b66441abbb98f928d8b30ec in term 0.
I20260925 22:46:19.881693 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.876534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:52835 (local address 127.0.201.130:38673)
0925 22:46:19.876795 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.876799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.876929 (+ 130us) server_negotiation.cc:407] Connection header received
0925 22:46:19.877024 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.877028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.877083 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.877174 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:19.878509 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.879247 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.880404 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.880991 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.881113 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.881162 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.881492 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.881566 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260925 22:46:19.881840 8426 leader_election.cc:302] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae706988b66441abbb98f928d8b30ec, c0f0bf025fef469a829de5257cd8c0cf; no voters:
I20260925 22:46:19.882107 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.876341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:38673 (local address 127.0.201.129:52835)
0925 22:46:19.876833 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.876845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.876931 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.877225 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.877230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.877250 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:19.877467 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.877474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.879377 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.879381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.880219 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.880229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.881223 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.881226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.881332 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.881355 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.881889 (+ 534us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.881963 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":186,"spinlock_wait_cycles":42240,"thread_start_us":108,"threads_started":1}
I20260925 22:46:19.882109 8855 raft_consensus.cc:2781] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:19.882231 8855 raft_consensus.cc:482] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:19.882280 8855 raft_consensus.cc:3037] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:19.882603 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358ac482e1b241cc976fb326fb04ec0c" candidate_uuid: "6ae706988b66441abbb98f928d8b30ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89f55716f8847a3a8959047a7dd7a35" is_pre_election: true
I20260925 22:46:19.882814 8635 raft_consensus.cc:2454] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae706988b66441abbb98f928d8b30ec in term 0.
I20260925 22:46:19.883524 8855 raft_consensus.cc:504] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.883950 8855 leader_election.cc:288] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [CANDIDATE]: Term 1 election: Requested vote from peers b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130:38673), c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:19.884264 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358ac482e1b241cc976fb326fb04ec0c" candidate_uuid: "6ae706988b66441abbb98f928d8b30ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89f55716f8847a3a8959047a7dd7a35"
I20260925 22:46:19.884323 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358ac482e1b241cc976fb326fb04ec0c" candidate_uuid: "6ae706988b66441abbb98f928d8b30ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f0bf025fef469a829de5257cd8c0cf"
I20260925 22:46:19.884384 8635 raft_consensus.cc:3037] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:19.884461 8774 raft_consensus.cc:3037] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:19.885354 8774 raft_consensus.cc:2454] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae706988b66441abbb98f928d8b30ec in term 1.
I20260925 22:46:19.885583 8635 raft_consensus.cc:2454] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae706988b66441abbb98f928d8b30ec in term 1.
I20260925 22:46:19.885617 8426 leader_election.cc:302] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae706988b66441abbb98f928d8b30ec, c0f0bf025fef469a829de5257cd8c0cf; no voters:
I20260925 22:46:19.885922 8855 raft_consensus.cc:2781] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:19.886133 8855 raft_consensus.cc:686] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 1 LEADER]: Becoming Leader. State: Replica: 6ae706988b66441abbb98f928d8b30ec, State: Running, Role: LEADER
I20260925 22:46:19.886377 8855 consensus_queue.cc:237] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.889004 8338 catalog_manager.cc:5495] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ae706988b66441abbb98f928d8b30ec (127.0.201.129). New cstate: current_term: 1 leader_uuid: "6ae706988b66441abbb98f928d8b30ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:46:19 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376378, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.190@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:19.928886 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.921332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40195 (local address 127.0.0.1:57322)
0925 22:46:19.921625 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:19.921643 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:19.921769 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:19.922018 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:19.922020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:19.922383 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:19.922474 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.922481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.923158 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.923160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.923766 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:19.923774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.924146 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.924149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:19.924205 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.924783 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:19.924795 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:19.926140 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:19.927525 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.927532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.927537 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.927817 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.928043 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:19.928045 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:19.928046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:19.928120 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:19.928408 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:19.928412 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:19.928507 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 22:46:19.928616 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 22:46:19.928728 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260925 22:46:19.928901 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:19.921464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.201.190:40195)
0925 22:46:19.921641 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:19.921644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:19.921723 (+ 79us) server_negotiation.cc:407] Connection header received
0925 22:46:19.921861 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:19.921863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:19.921901 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:19.921971 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:19.922573 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.923070 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.923860 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:19.924056 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:19.924129 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:19.926254 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:19.926257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:19.926259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:19.926279 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:19.927432 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.927900 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.927903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.927904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.927970 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:19.928176 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:19.928178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:19.928179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:19.928331 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:19.928395 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:19.928665 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 22:46:19.928780 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":109}
I20260925 22:46:19.932842 8338 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:57322:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:46:19.933481 8338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:19.938122 8475 tablet_service.cc:1467] Processing CreateTablet for tablet 226a872b52854a938a40294495bf76be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a55b3c36b4a04592bd2628f99ff458c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:46:19.938107 8615 tablet_service.cc:1467] Processing CreateTablet for tablet 226a872b52854a938a40294495bf76be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a55b3c36b4a04592bd2628f99ff458c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:46:19.938341 8755 tablet_service.cc:1467] Processing CreateTablet for tablet 226a872b52854a938a40294495bf76be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a55b3c36b4a04592bd2628f99ff458c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:46:19.938503 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226a872b52854a938a40294495bf76be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:19.938503 8475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226a872b52854a938a40294495bf76be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:19.938724 8755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226a872b52854a938a40294495bf76be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:19.941498 8849 tablet_bootstrap.cc:492] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: Bootstrap starting.
I20260925 22:46:19.941519 8851 tablet_bootstrap.cc:492] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: Bootstrap starting.
I20260925 22:46:19.941910 8850 tablet_bootstrap.cc:492] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: Bootstrap starting.
I20260925 22:46:19.942402 8849 tablet_bootstrap.cc:654] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:19.942428 8851 tablet_bootstrap.cc:654] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:19.942823 8850 tablet_bootstrap.cc:654] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:19.943555 8849 tablet_bootstrap.cc:492] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: No bootstrap required, opened a new log
I20260925 22:46:19.943663 8849 ts_tablet_manager.cc:1397] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:19.943768 8851 tablet_bootstrap.cc:492] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: No bootstrap required, opened a new log
I20260925 22:46:19.943859 8851 ts_tablet_manager.cc:1397] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:19.944036 8850 tablet_bootstrap.cc:492] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: No bootstrap required, opened a new log
I20260925 22:46:19.944115 8850 ts_tablet_manager.cc:1397] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:46:19.944160 8849 raft_consensus.cc:348] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.944299 8849 raft_consensus.cc:374] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:19.944281 8851 raft_consensus.cc:348] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.944339 8849 raft_consensus.cc:729] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f0bf025fef469a829de5257cd8c0cf, State: Initialized, Role: FOLLOWER
I20260925 22:46:19.944389 8851 raft_consensus.cc:374] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:19.944430 8851 raft_consensus.cc:729] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89f55716f8847a3a8959047a7dd7a35, State: Initialized, Role: FOLLOWER
I20260925 22:46:19.944480 8849 consensus_queue.cc:260] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.944545 8851 consensus_queue.cc:260] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.944541 8850 raft_consensus.cc:348] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.944633 8850 raft_consensus.cc:374] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:19.944689 8850 raft_consensus.cc:729] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae706988b66441abbb98f928d8b30ec, State: Initialized, Role: FOLLOWER
I20260925 22:46:19.944800 8849 ts_tablet_manager.cc:1428] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:19.944810 8851 ts_tablet_manager.cc:1428] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:19.944810 8850 consensus_queue.cc:260] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:19.945063 8850 ts_tablet_manager.cc:1428] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:19.945020 8867 tablet_replica.cc:439] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } }
I20260925 22:46:19.945142 8867 tablet_replica.cc:442] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:46:19.945149 8856 tablet_replica.cc:439] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } }
I20260925 22:46:19.945230 8855 tablet_replica.cc:439] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } }
I20260925 22:46:19.945286 8856 tablet_replica.cc:442] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:46:19.945333 8855 tablet_replica.cc:442] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 22:46:20.006155 8682 tablet.cc:2369] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:20.093387 8855 raft_consensus.cc:482] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:20.093532 8855 raft_consensus.cc:504] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.094023 8855 leader_election.cc:288] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130:38673), c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:20.094507 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226a872b52854a938a40294495bf76be" candidate_uuid: "6ae706988b66441abbb98f928d8b30ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89f55716f8847a3a8959047a7dd7a35" is_pre_election: true
I20260925 22:46:20.094529 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226a872b52854a938a40294495bf76be" candidate_uuid: "6ae706988b66441abbb98f928d8b30ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f0bf025fef469a829de5257cd8c0cf" is_pre_election: true
I20260925 22:46:20.094689 8774 raft_consensus.cc:2454] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae706988b66441abbb98f928d8b30ec in term 0.
I20260925 22:46:20.094683 8635 raft_consensus.cc:2454] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae706988b66441abbb98f928d8b30ec in term 0.
I20260925 22:46:20.095028 8428 leader_election.cc:302] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae706988b66441abbb98f928d8b30ec, b89f55716f8847a3a8959047a7dd7a35; no voters:
I20260925 22:46:20.095204 8855 raft_consensus.cc:2781] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:20.095268 8855 raft_consensus.cc:482] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:20.095311 8855 raft_consensus.cc:3037] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:20.096292 8855 raft_consensus.cc:504] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.096661 8855 leader_election.cc:288] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [CANDIDATE]: Term 1 election: Requested vote from peers b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130:38673), c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:20.096971 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226a872b52854a938a40294495bf76be" candidate_uuid: "6ae706988b66441abbb98f928d8b30ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89f55716f8847a3a8959047a7dd7a35"
I20260925 22:46:20.097110 8635 raft_consensus.cc:3037] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:20.097105 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226a872b52854a938a40294495bf76be" candidate_uuid: "6ae706988b66441abbb98f928d8b30ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f0bf025fef469a829de5257cd8c0cf"
I20260925 22:46:20.097241 8774 raft_consensus.cc:3037] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:20.098091 8635 raft_consensus.cc:2454] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae706988b66441abbb98f928d8b30ec in term 1.
I20260925 22:46:20.098140 8774 raft_consensus.cc:2454] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae706988b66441abbb98f928d8b30ec in term 1.
I20260925 22:46:20.098392 8428 leader_election.cc:302] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae706988b66441abbb98f928d8b30ec, b89f55716f8847a3a8959047a7dd7a35; no voters:
I20260925 22:46:20.098585 8855 raft_consensus.cc:2781] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:20.098672 8855 raft_consensus.cc:686] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [term 1 LEADER]: Becoming Leader. State: Replica: 6ae706988b66441abbb98f928d8b30ec, State: Running, Role: LEADER
I20260925 22:46:20.098798 8855 consensus_queue.cc:237] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.099503 8880 tablet_replica.cc:439] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: TxnStatusTablet state changed. Reason: New leader 6ae706988b66441abbb98f928d8b30ec. Latest consensus state: current_term: 1 leader_uuid: "6ae706988b66441abbb98f928d8b30ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } }
I20260925 22:46:20.099606 8880 tablet_replica.cc:442] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:46:20.099962 8892 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:46:20.100145 8892 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:46:20.100548 8336 catalog_manager.cc:5495] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ae706988b66441abbb98f928d8b30ec (127.0.201.129). New cstate: current_term: 1 leader_uuid: "6ae706988b66441abbb98f928d8b30ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } health_report { overall_health: UNKNOWN } } }
I20260925 22:46:20.135730 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.131850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38922 (local address 127.0.201.129:46151)
0925 22:46:20.131984 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:20.131987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:20.132023 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:46:20.132237 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:20.132240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:20.132290 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:20.132373 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:20.132635 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.133137 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.134332 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.134533 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.134605 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.134918 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:20.135317 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:20.135401 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:20.135512 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 22:46:20.135560 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:46:20.135751 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.131795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:46151 (local address 127.0.0.1:38922)
0925 22:46:20.131954 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:20.131970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:20.132094 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:20.132406 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:20.132409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:20.132416 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:46:20.132532 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:20.132538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.133246 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.133249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:20.134219 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:20.134229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.134631 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.134635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:20.134730 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.134782 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:46:20.135408 (+ 626us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:46:20.135421 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 22:46:20.135498 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:46:20.135584 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 22:46:20.139114 8774 raft_consensus.cc:1261] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Refusing update from remote peer 6ae706988b66441abbb98f928d8b30ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:20.139310 8635 raft_consensus.cc:1261] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Refusing update from remote peer 6ae706988b66441abbb98f928d8b30ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:20.139637 8855 consensus_queue.cc:1035] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:20.139957 8880 consensus_queue.cc:1035] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:20.141522 8867 tablet_replica.cc:439] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: TxnStatusTablet state changed. Reason: New leader 6ae706988b66441abbb98f928d8b30ec. Latest consensus state: current_term: 1 leader_uuid: "6ae706988b66441abbb98f928d8b30ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } }
I20260925 22:46:20.141640 8867 tablet_replica.cc:442] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:46:20.142828 8856 tablet_replica.cc:439] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: TxnStatusTablet state changed. Reason: New leader 6ae706988b66441abbb98f928d8b30ec. Latest consensus state: current_term: 1 leader_uuid: "6ae706988b66441abbb98f928d8b30ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } }
I20260925 22:46:20.142940 8856 tablet_replica.cc:442] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:46:20.142944 8880 tablet_replica.cc:439] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ae706988b66441abbb98f928d8b30ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } }
I20260925 22:46:20.143051 8880 tablet_replica.cc:442] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:46:20.144773 8867 tablet_replica.cc:439] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ae706988b66441abbb98f928d8b30ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } }
I20260925 22:46:20.144881 8867 tablet_replica.cc:442] T 226a872b52854a938a40294495bf76be P b89f55716f8847a3a8959047a7dd7a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:46:20.146174 8855 tablet_replica.cc:439] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ae706988b66441abbb98f928d8b30ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } }
I20260925 22:46:20.146266 8855 tablet_replica.cc:442] T 226a872b52854a938a40294495bf76be P 6ae706988b66441abbb98f928d8b30ec: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:46:20.147755 8864 mvcc.cc:205] Tried to move back new op lower bound from 7333381653043011584 to 7333381652886122496. Current Snapshot: MvccSnapshot[applied={T|T < 7333381653043011584 or (T in {7333381653043011584})}]
I20260925 22:46:20.148094 8856 tablet_replica.cc:439] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ae706988b66441abbb98f928d8b30ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } } }
I20260925 22:46:20.148191 8856 tablet_replica.cc:442] T 226a872b52854a938a40294495bf76be P c0f0bf025fef469a829de5257cd8c0cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:46:20.151464 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.148176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.0.201.129:46151)
0925 22:46:20.148281 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:20.148285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:20.148504 (+ 219us) server_negotiation.cc:407] Connection header received
0925 22:46:20.148540 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:20.148541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:20.148576 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:20.148657 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:20.148909 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.149503 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.150465 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.150693 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.150805 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.151044 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:20.151113 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:20.151195 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:20.151289 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 22:46:20.151334 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:46:20.161811 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.158279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:46151 (local address 127.0.201.129:58573)
0925 22:46:20.158559 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:20.158570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:20.158678 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:20.158911 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:20.158914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:20.158920 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:46:20.159049 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:20.159059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.159784 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.159790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:20.160558 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:20.160566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.161081 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.161085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:20.161178 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.161224 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:46:20.161537 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:46:20.161542 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:46:20.161590 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 22:46:20.161647 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":23,"thread_start_us":86,"threads_started":1}
I20260925 22:46:20.161810 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.158358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:58573 (local address 127.0.201.129:46151)
0925 22:46:20.158470 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:20.158476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:20.158639 (+ 163us) server_negotiation.cc:407] Connection header received
0925 22:46:20.158774 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:20.158777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:20.158817 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:20.158895 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:20.159152 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.159662 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.160689 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.160939 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.161052 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.161374 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:20.161450 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:20.161528 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:20.161600 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 22:46:20.161649 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":41}
I20260925 22:46:20.167796 8774 raft_consensus.cc:1261] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Refusing update from remote peer 6ae706988b66441abbb98f928d8b30ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:20.167824 8635 raft_consensus.cc:1261] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Refusing update from remote peer 6ae706988b66441abbb98f928d8b30ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:20.168159 8880 consensus_queue.cc:1035] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:20.168354 8855 consensus_queue.cc:1035] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:20.203743 8336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:20.203904 8336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:20.205794 8336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103: Sending DeleteTablet for 3 replicas of tablet 358ac482e1b241cc976fb326fb04ec0c
I20260925 22:46:20.206360 8615 tablet_service.cc:1514] Processing DeleteTablet for tablet 358ac482e1b241cc976fb326fb04ec0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:20 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:57380
I20260925 22:46:20.206521 8475 tablet_service.cc:1514] Processing DeleteTablet for tablet 358ac482e1b241cc976fb326fb04ec0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:20 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:38918
I20260925 22:46:20.206549 8755 tablet_service.cc:1514] Processing DeleteTablet for tablet 358ac482e1b241cc976fb326fb04ec0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:20 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:53252
I20260925 22:46:20.206714 8872 tablet_replica.cc:330] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: stopping tablet replica
I20260925 22:46:20.206732 8874 tablet_replica.cc:330] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf: stopping tablet replica
I20260925 22:46:20.206740 8873 tablet_replica.cc:330] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35: stopping tablet replica
I20260925 22:46:20.206820 8872 raft_consensus.cc:2229] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:46:20.206835 8874 raft_consensus.cc:2229] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:20.206843 8873 raft_consensus.cc:2229] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:20.206928 8874 raft_consensus.cc:2258] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:20.206934 8873 raft_consensus.cc:2258] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:20.207083 8872 raft_consensus.cc:2258] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:20.210680 8873 ts_tablet_manager.cc:1905] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:20.212164 8874 ts_tablet_manager.cc:1905] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:20.213029 8873 ts_tablet_manager.cc:1918] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:46:20.213120 8873 log.cc:1196] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/358ac482e1b241cc976fb326fb04ec0c
I20260925 22:46:20.213359 8873 ts_tablet_manager.cc:1939] T 358ac482e1b241cc976fb326fb04ec0c P b89f55716f8847a3a8959047a7dd7a35: Deleting consensus metadata
I20260925 22:46:20.214041 8324 catalog_manager.cc:4841] TS b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130:38673): tablet 358ac482e1b241cc976fb326fb04ec0c (table test-table [id=1e995b210b954cdebd0c0c06b96d2354]) successfully deleted
I20260925 22:46:20.214794 8874 ts_tablet_manager.cc:1918] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:46:20.214866 8874 log.cc:1196] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/358ac482e1b241cc976fb326fb04ec0c
I20260925 22:46:20.215116 8874 ts_tablet_manager.cc:1939] T 358ac482e1b241cc976fb326fb04ec0c P c0f0bf025fef469a829de5257cd8c0cf: Deleting consensus metadata
I20260925 22:46:20.215904 8322 catalog_manager.cc:4841] TS c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649): tablet 358ac482e1b241cc976fb326fb04ec0c (table test-table [id=1e995b210b954cdebd0c0c06b96d2354]) successfully deleted
I20260925 22:46:20.216220 8872 ts_tablet_manager.cc:1905] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:20.218520 8872 ts_tablet_manager.cc:1918] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:46:20.218606 8872 log.cc:1196] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/358ac482e1b241cc976fb326fb04ec0c
I20260925 22:46:20.218825 8872 ts_tablet_manager.cc:1939] T 358ac482e1b241cc976fb326fb04ec0c P 6ae706988b66441abbb98f928d8b30ec: Deleting consensus metadata
I20260925 22:46:20.219466 8325 catalog_manager.cc:4841] TS 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151): tablet 358ac482e1b241cc976fb326fb04ec0c (table test-table [id=1e995b210b954cdebd0c0c06b96d2354]) successfully deleted
Sep 25 22:46:20 dist-test-slave-vdtf krb5kdc[8293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:46:20 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:20 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:46:20.243423 8387 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:20.236653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.0.201.190:40195)
0925 22:46:20.236778 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:20.236783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:20.236937 (+ 154us) server_negotiation.cc:407] Connection header received
0925 22:46:20.237051 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:20.237054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:20.237109 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:20.237214 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:20.237950 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.238745 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.239626 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.239977 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.240102 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.243299 (+ 3197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57330: BlockingRecv error: recv got EOF from 127.0.0.1:57330 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 22:46:20 dist-test-slave-vdtf krb5kdc[8293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM
I20260925 22:46:20.253592 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.244789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57332 (local address 127.0.201.190:40195)
0925 22:46:20.244920 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:20.244924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:20.245164 (+ 240us) server_negotiation.cc:407] Connection header received
0925 22:46:20.245268 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:20.245271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:20.245306 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:20.245370 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:20.246125 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.246585 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.247734 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.247928 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.248003 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.250509 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:20.250512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:20.250515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:20.250538 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:20.252022 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:20.252423 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:20.252428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:20.252430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:20.252510 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:20.252783 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:20.252786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:20.252788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:20.252955 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:20.253088 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:20.253319 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 22:46:20.253433 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:46:20.256253 8336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:46:20.256726 8336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:20.261567 8475 tablet_service.cc:1467] Processing CreateTablet for tablet 6425d51989014c60879b6498b0c2bbe8 (DEFAULT_TABLE table=test-table [id=d4069a67a0b14dbd8e766b2e227bbc26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:20.261657 8615 tablet_service.cc:1467] Processing CreateTablet for tablet 6425d51989014c60879b6498b0c2bbe8 (DEFAULT_TABLE table=test-table [id=d4069a67a0b14dbd8e766b2e227bbc26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:20.261844 8475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6425d51989014c60879b6498b0c2bbe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:20.261940 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6425d51989014c60879b6498b0c2bbe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:20.264015 8851 tablet_bootstrap.cc:492] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35: Bootstrap starting.
I20260925 22:46:20.264009 8755 tablet_service.cc:1467] Processing CreateTablet for tablet 6425d51989014c60879b6498b0c2bbe8 (DEFAULT_TABLE table=test-table [id=d4069a67a0b14dbd8e766b2e227bbc26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:20.264149 8850 tablet_bootstrap.cc:492] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec: Bootstrap starting.
I20260925 22:46:20.264261 8755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6425d51989014c60879b6498b0c2bbe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:20.264837 8851 tablet_bootstrap.cc:654] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:20.265014 8850 tablet_bootstrap.cc:654] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:20.266459 8849 tablet_bootstrap.cc:492] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf: Bootstrap starting.
I20260925 22:46:20.267373 8849 tablet_bootstrap.cc:654] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:20.268503 8850 tablet_bootstrap.cc:492] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec: No bootstrap required, opened a new log
I20260925 22:46:20.268599 8850 ts_tablet_manager.cc:1397] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:46:20.268790 8851 tablet_bootstrap.cc:492] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35: No bootstrap required, opened a new log
I20260925 22:46:20.268873 8851 ts_tablet_manager.cc:1397] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 22:46:20.269003 8850 raft_consensus.cc:348] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.269115 8850 raft_consensus.cc:374] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:20.269145 8850 raft_consensus.cc:729] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae706988b66441abbb98f928d8b30ec, State: Initialized, Role: FOLLOWER
I20260925 22:46:20.269248 8850 consensus_queue.cc:260] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.269292 8851 raft_consensus.cc:348] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.269397 8851 raft_consensus.cc:374] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:20.269435 8851 raft_consensus.cc:729] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89f55716f8847a3a8959047a7dd7a35, State: Initialized, Role: FOLLOWER
I20260925 22:46:20.269479 8850 ts_tablet_manager.cc:1428] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:20.269553 8851 consensus_queue.cc:260] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.269816 8851 ts_tablet_manager.cc:1428] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:20.273166 8849 tablet_bootstrap.cc:492] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf: No bootstrap required, opened a new log
I20260925 22:46:20.273286 8849 ts_tablet_manager.cc:1397] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 22:46:20.273695 8849 raft_consensus.cc:348] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.273814 8849 raft_consensus.cc:374] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:20.273854 8849 raft_consensus.cc:729] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f0bf025fef469a829de5257cd8c0cf, State: Initialized, Role: FOLLOWER
I20260925 22:46:20.273993 8849 consensus_queue.cc:260] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.274289 8849 ts_tablet_manager.cc:1428] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:46:20.288729 8867 raft_consensus.cc:482] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:20.288870 8867 raft_consensus.cc:504] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.289314 8867 leader_election.cc:288] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151), c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:20.289716 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6425d51989014c60879b6498b0c2bbe8" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f0bf025fef469a829de5257cd8c0cf" is_pre_election: true
I20260925 22:46:20.289724 8492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6425d51989014c60879b6498b0c2bbe8" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae706988b66441abbb98f928d8b30ec" is_pre_election: true
I20260925 22:46:20.289865 8492 raft_consensus.cc:2454] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 0.
I20260925 22:46:20.289867 8774 raft_consensus.cc:2454] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 0.
I20260925 22:46:20.290211 8566 leader_election.cc:302] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b89f55716f8847a3a8959047a7dd7a35, c0f0bf025fef469a829de5257cd8c0cf; no voters:
I20260925 22:46:20.290378 8867 raft_consensus.cc:2781] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:20.290423 8867 raft_consensus.cc:482] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:20.290457 8867 raft_consensus.cc:3037] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:20.291646 8867 raft_consensus.cc:504] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.292016 8867 leader_election.cc:288] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 election: Requested vote from peers 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151), c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:20.292292 8492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6425d51989014c60879b6498b0c2bbe8" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae706988b66441abbb98f928d8b30ec"
I20260925 22:46:20.292441 8492 raft_consensus.cc:3037] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:20.292483 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6425d51989014c60879b6498b0c2bbe8" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f0bf025fef469a829de5257cd8c0cf"
I20260925 22:46:20.292598 8774 raft_consensus.cc:3037] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:20.293524 8492 raft_consensus.cc:2454] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 1.
I20260925 22:46:20.293673 8774 raft_consensus.cc:2454] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 1.
I20260925 22:46:20.293821 8569 leader_election.cc:302] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae706988b66441abbb98f928d8b30ec, b89f55716f8847a3a8959047a7dd7a35; no voters:
I20260925 22:46:20.293982 8867 raft_consensus.cc:2781] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:20.294148 8867 raft_consensus.cc:686] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 1 LEADER]: Becoming Leader. State: Replica: b89f55716f8847a3a8959047a7dd7a35, State: Running, Role: LEADER
I20260925 22:46:20.294304 8867 consensus_queue.cc:237] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.295996 8337 catalog_manager.cc:5495] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130). New cstate: current_term: 1 leader_uuid: "b89f55716f8847a3a8959047a7dd7a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } health_report { overall_health: UNKNOWN } } }
I20260925 22:46:20.312176 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.308919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.0.201.130:38673)
0925 22:46:20.309058 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:20.309063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:20.309078 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:46:20.309166 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:20.309169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:20.309217 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:20.309297 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:20.309497 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.310282 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.311063 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.311399 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.311501 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.311745 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:20.311826 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:20.311899 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:20.311979 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:46:20.312039 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:46:20.314122 8492 raft_consensus.cc:1261] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Refusing update from remote peer b89f55716f8847a3a8959047a7dd7a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:20.314122 8774 raft_consensus.cc:1261] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Refusing update from remote peer b89f55716f8847a3a8959047a7dd7a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:20.314502 8920 consensus_queue.cc:1035] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:20.314685 8867 consensus_queue.cc:1035] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:20.322960 8338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:20.323113 8338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:20.325183 8338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103: Sending DeleteTablet for 3 replicas of tablet 6425d51989014c60879b6498b0c2bbe8
I20260925 22:46:20.325665 8475 tablet_service.cc:1514] Processing DeleteTablet for tablet 6425d51989014c60879b6498b0c2bbe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:20 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:38918
I20260925 22:46:20.325794 8615 tablet_service.cc:1514] Processing DeleteTablet for tablet 6425d51989014c60879b6498b0c2bbe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:20 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:57380
I20260925 22:46:20.325838 8872 tablet_replica.cc:330] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec: stopping tablet replica
I20260925 22:46:20.325892 8755 tablet_service.cc:1514] Processing DeleteTablet for tablet 6425d51989014c60879b6498b0c2bbe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:20 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:53252
I20260925 22:46:20.325946 8873 tablet_replica.cc:330] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35: stopping tablet replica
I20260925 22:46:20.325951 8872 raft_consensus.cc:2229] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:20.326035 8874 tablet_replica.cc:330] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf: stopping tablet replica
I20260925 22:46:20.326052 8873 raft_consensus.cc:2229] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:46:20.326066 8872 raft_consensus.cc:2258] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:20.326131 8874 raft_consensus.cc:2229] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:20.326201 8873 raft_consensus.cc:2258] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:20.326242 8874 raft_consensus.cc:2258] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:20.326934 8873 ts_tablet_manager.cc:1905] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:20.327082 8872 ts_tablet_manager.cc:1905] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:20.327018 8338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:46:20.327366 8338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:20.329437 8873 ts_tablet_manager.cc:1918] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:20.329514 8873 log.cc:1196] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/6425d51989014c60879b6498b0c2bbe8
I20260925 22:46:20.329536 8872 ts_tablet_manager.cc:1918] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:20.329602 8872 log.cc:1196] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/6425d51989014c60879b6498b0c2bbe8
I20260925 22:46:20.329584 8874 ts_tablet_manager.cc:1905] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:20.329775 8873 ts_tablet_manager.cc:1939] T 6425d51989014c60879b6498b0c2bbe8 P b89f55716f8847a3a8959047a7dd7a35: Deleting consensus metadata
I20260925 22:46:20.329881 8872 ts_tablet_manager.cc:1939] T 6425d51989014c60879b6498b0c2bbe8 P 6ae706988b66441abbb98f928d8b30ec: Deleting consensus metadata
I20260925 22:46:20.330410 8325 catalog_manager.cc:4841] TS 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151): tablet 6425d51989014c60879b6498b0c2bbe8 (table test-table [id=d4069a67a0b14dbd8e766b2e227bbc26]) successfully deleted
I20260925 22:46:20.331795 8324 catalog_manager.cc:4841] TS b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130:38673): tablet 6425d51989014c60879b6498b0c2bbe8 (table test-table [id=d4069a67a0b14dbd8e766b2e227bbc26]) successfully deleted
I20260925 22:46:20.331799 8874 ts_tablet_manager.cc:1918] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:20.331887 8874 log.cc:1196] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/6425d51989014c60879b6498b0c2bbe8
I20260925 22:46:20.332140 8874 ts_tablet_manager.cc:1939] T 6425d51989014c60879b6498b0c2bbe8 P c0f0bf025fef469a829de5257cd8c0cf: Deleting consensus metadata
I20260925 22:46:20.332790 8322 catalog_manager.cc:4841] TS c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649): tablet 6425d51989014c60879b6498b0c2bbe8 (table test-table [id=d4069a67a0b14dbd8e766b2e227bbc26]) successfully deleted
I20260925 22:46:20.334432 8475 tablet_service.cc:1467] Processing CreateTablet for tablet 826dc578e5094d979b36aa69941b2d9a (DEFAULT_TABLE table=test-table [id=119561da650440cc8d8e316f8526f2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:20.334555 8615 tablet_service.cc:1467] Processing CreateTablet for tablet 826dc578e5094d979b36aa69941b2d9a (DEFAULT_TABLE table=test-table [id=119561da650440cc8d8e316f8526f2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:20.334661 8475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 826dc578e5094d979b36aa69941b2d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:20.334651 8755 tablet_service.cc:1467] Processing CreateTablet for tablet 826dc578e5094d979b36aa69941b2d9a (DEFAULT_TABLE table=test-table [id=119561da650440cc8d8e316f8526f2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:20.334781 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 826dc578e5094d979b36aa69941b2d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:20.334865 8755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 826dc578e5094d979b36aa69941b2d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:20.336748 8850 tablet_bootstrap.cc:492] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec: Bootstrap starting.
I20260925 22:46:20.336943 8849 tablet_bootstrap.cc:492] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf: Bootstrap starting.
I20260925 22:46:20.336943 8851 tablet_bootstrap.cc:492] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35: Bootstrap starting.
I20260925 22:46:20.337534 8850 tablet_bootstrap.cc:654] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:20.337719 8851 tablet_bootstrap.cc:654] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:20.337762 8849 tablet_bootstrap.cc:654] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:20.338979 8849 tablet_bootstrap.cc:492] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf: No bootstrap required, opened a new log
I20260925 22:46:20.339066 8849 ts_tablet_manager.cc:1397] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:46:20.339094 8850 tablet_bootstrap.cc:492] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec: No bootstrap required, opened a new log
I20260925 22:46:20.339172 8850 ts_tablet_manager.cc:1397] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:46:20.339500 8851 tablet_bootstrap.cc:492] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35: No bootstrap required, opened a new log
I20260925 22:46:20.339476 8849 raft_consensus.cc:348] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.339573 8849 raft_consensus.cc:374] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:20.339573 8851 ts_tablet_manager.cc:1397] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:46:20.339604 8849 raft_consensus.cc:729] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f0bf025fef469a829de5257cd8c0cf, State: Initialized, Role: FOLLOWER
I20260925 22:46:20.339562 8850 raft_consensus.cc:348] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.339686 8850 raft_consensus.cc:374] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:20.339725 8850 raft_consensus.cc:729] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae706988b66441abbb98f928d8b30ec, State: Initialized, Role: FOLLOWER
I20260925 22:46:20.339728 8849 consensus_queue.cc:260] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.339833 8850 consensus_queue.cc:260] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.339970 8849 ts_tablet_manager.cc:1428] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:20.339978 8851 raft_consensus.cc:348] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.340068 8851 raft_consensus.cc:374] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:20.340092 8850 ts_tablet_manager.cc:1428] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:20.340101 8851 raft_consensus.cc:729] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89f55716f8847a3a8959047a7dd7a35, State: Initialized, Role: FOLLOWER
I20260925 22:46:20.340226 8851 consensus_queue.cc:260] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.340530 8851 ts_tablet_manager.cc:1428] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:46:20.616092 8920 raft_consensus.cc:482] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:20.616269 8920 raft_consensus.cc:504] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.616752 8920 leader_election.cc:288] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151), c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:20.617215 8492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826dc578e5094d979b36aa69941b2d9a" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae706988b66441abbb98f928d8b30ec" is_pre_election: true
I20260925 22:46:20.617215 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826dc578e5094d979b36aa69941b2d9a" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f0bf025fef469a829de5257cd8c0cf" is_pre_election: true
I20260925 22:46:20.617340 8492 raft_consensus.cc:2454] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 0.
I20260925 22:46:20.617417 8774 raft_consensus.cc:2454] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 0.
I20260925 22:46:20.617697 8569 leader_election.cc:302] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae706988b66441abbb98f928d8b30ec, b89f55716f8847a3a8959047a7dd7a35; no voters:
I20260925 22:46:20.617883 8920 raft_consensus.cc:2781] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:20.617969 8920 raft_consensus.cc:482] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:20.618026 8920 raft_consensus.cc:3037] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:20.619127 8920 raft_consensus.cc:504] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.619515 8920 leader_election.cc:288] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 election: Requested vote from peers 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151), c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649)
I20260925 22:46:20.619879 8492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826dc578e5094d979b36aa69941b2d9a" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae706988b66441abbb98f928d8b30ec"
I20260925 22:46:20.619944 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826dc578e5094d979b36aa69941b2d9a" candidate_uuid: "b89f55716f8847a3a8959047a7dd7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f0bf025fef469a829de5257cd8c0cf"
I20260925 22:46:20.620023 8492 raft_consensus.cc:3037] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:20.620077 8774 raft_consensus.cc:3037] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:20.620981 8492 raft_consensus.cc:2454] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 1.
I20260925 22:46:20.621042 8774 raft_consensus.cc:2454] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89f55716f8847a3a8959047a7dd7a35 in term 1.
I20260925 22:46:20.621330 8566 leader_election.cc:302] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6ae706988b66441abbb98f928d8b30ec, b89f55716f8847a3a8959047a7dd7a35, c0f0bf025fef469a829de5257cd8c0cf; no voters:
I20260925 22:46:20.621534 8920 raft_consensus.cc:2781] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:20.621644 8920 raft_consensus.cc:686] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 1 LEADER]: Becoming Leader. State: Replica: b89f55716f8847a3a8959047a7dd7a35, State: Running, Role: LEADER
I20260925 22:46:20.621804 8920 consensus_queue.cc:237] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } }
I20260925 22:46:20.623425 8337 catalog_manager.cc:5495] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130). New cstate: current_term: 1 leader_uuid: "b89f55716f8847a3a8959047a7dd7a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89f55716f8847a3a8959047a7dd7a35" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 38673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 } health_report { overall_health: UNKNOWN } } }
I20260925 22:46:20.650142 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.647347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:56295 (local address 127.0.201.129:46151)
0925 22:46:20.647474 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:20.647478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:20.647623 (+ 145us) server_negotiation.cc:407] Connection header received
0925 22:46:20.647740 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:20.647743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:20.647783 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:20.647847 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:20.648040 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.648521 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.649308 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.649507 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.649580 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.649810 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:20.649870 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:20.649921 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:20.649997 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:46:20.650033 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:46:20.650158 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.647277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:46151 (local address 127.0.201.130:56295)
0925 22:46:20.647545 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:20.647558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:20.647647 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:20.647866 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:20.647867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:20.647873 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:46:20.647961 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:20.647965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.648600 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.648603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:20.649212 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:20.649220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.649595 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.649598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:20.649658 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.649706 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:46:20.649935 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:46:20.649940 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:46:20.649983 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 22:46:20.650050 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260925 22:46:20.655232 8774 raft_consensus.cc:1261] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Refusing update from remote peer b89f55716f8847a3a8959047a7dd7a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:20.655226 8492 raft_consensus.cc:1261] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Refusing update from remote peer b89f55716f8847a3a8959047a7dd7a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:20.655629 8920 consensus_queue.cc:1035] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f0bf025fef469a829de5257cd8c0cf" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:20.655850 8925 consensus_queue.cc:1035] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae706988b66441abbb98f928d8b30ec" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:20.671389 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.668242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:42461 (local address 127.0.201.130:38673)
0925 22:46:20.668366 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:20.668371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:20.668383 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:46:20.668422 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:20.668426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:20.668480 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:20.668578 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:20.668819 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.669360 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.670255 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.670555 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.670668 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.670959 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:20.671038 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:20.671110 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:20.671211 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 22:46:20.671261 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:46:20.671397 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.667831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:38673 (local address 127.0.201.129:42461)
0925 22:46:20.668041 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:20.668054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:20.668125 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:20.668592 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:20.668595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:20.668603 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:46:20.668710 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:20.668716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.669469 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.669474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:20.670139 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:20.670148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:20.670692 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:20.670696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:20.670801 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:20.670841 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:46:20.671124 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:46:20.671131 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:46:20.671197 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:46:20.671277 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260925 22:46:20.689756 8335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:20.689900 8335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:20.691732 8335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce5569581c1243b9afc87e91506d9103: Sending DeleteTablet for 3 replicas of tablet 826dc578e5094d979b36aa69941b2d9a
I20260925 22:46:20.692209 8475 tablet_service.cc:1514] Processing DeleteTablet for tablet 826dc578e5094d979b36aa69941b2d9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:20 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:38918
I20260925 22:46:20.692242 8615 tablet_service.cc:1514] Processing DeleteTablet for tablet 826dc578e5094d979b36aa69941b2d9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:20 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:57380
I20260925 22:46:20.692382 8872 tablet_replica.cc:330] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec: stopping tablet replica
I20260925 22:46:20.692382 8873 tablet_replica.cc:330] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35: stopping tablet replica
I20260925 22:46:20.692394 8755 tablet_service.cc:1514] Processing DeleteTablet for tablet 826dc578e5094d979b36aa69941b2d9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:20 UTC) from {username='oryx', principal='oryx/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:53252
I20260925 22:46:20.692478 8872 raft_consensus.cc:2229] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:20.692477 8873 raft_consensus.cc:2229] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:46:20.692574 8872 raft_consensus.cc:2258] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:20.692581 8874 tablet_replica.cc:330] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf: stopping tablet replica
I20260925 22:46:20.692655 8873 raft_consensus.cc:2258] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:20.692694 8874 raft_consensus.cc:2229] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:20.692804 8874 raft_consensus.cc:2258] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:20.693336 8872 ts_tablet_manager.cc:1905] T 826dc578e5094d979b36aa69941b2d9a P 6ae706988b66441abbb98f928d8b30ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:20.693387 8873 ts_tablet_manager.cc:1905] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:20.693506 8874 ts_tablet_manager.cc:1905] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:20.693830 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 8410
I20260925 22:46:20.696609 8874 ts_tablet_manager.cc:1918] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:46:20.696709 8874 log.cc:1196] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/826dc578e5094d979b36aa69941b2d9a
I20260925 22:46:20.696941 8874 ts_tablet_manager.cc:1939] T 826dc578e5094d979b36aa69941b2d9a P c0f0bf025fef469a829de5257cd8c0cf: Deleting consensus metadata
I20260925 22:46:20.697475 8873 ts_tablet_manager.cc:1918] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:46:20.697582 8873 log.cc:1196] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/826dc578e5094d979b36aa69941b2d9a
I20260925 22:46:20.697582 8322 catalog_manager.cc:4841] TS c0f0bf025fef469a829de5257cd8c0cf (127.0.201.131:36649): tablet 826dc578e5094d979b36aa69941b2d9a (table test-table [id=119561da650440cc8d8e316f8526f2a9]) successfully deleted
I20260925 22:46:20.697847 8873 ts_tablet_manager.cc:1939] T 826dc578e5094d979b36aa69941b2d9a P b89f55716f8847a3a8959047a7dd7a35: Deleting consensus metadata
I20260925 22:46:20.698395 8324 catalog_manager.cc:4841] TS b89f55716f8847a3a8959047a7dd7a35 (127.0.201.130:38673): tablet 826dc578e5094d979b36aa69941b2d9a (table test-table [id=119561da650440cc8d8e316f8526f2a9]) successfully deleted
W20260925 22:46:20.703630 8325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:46151 (error 108)
I20260925 22:46:20.703855 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 8550
I20260925 22:46:20.705914 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:20.705454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:46151 (local address 127.0.0.1:38934)
0925 22:46:20.705704 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:20.705786 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:46151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":50,"threads_started":1}
W20260925 22:46:20.706058 8325 catalog_manager.cc:4584] TS 6ae706988b66441abbb98f928d8b30ec (127.0.201.129:46151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 826dc578e5094d979b36aa69941b2d9a: Network error: Client connection negotiation failed: client connection to 127.0.201.129:46151: connect: Connection refused (error 111)
I20260925 22:46:20.710876 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 8690
I20260925 22:46:20.716356 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 8306
2026-09-25T22:46:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3324 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:46:20 dist-test-slave-vdtf krb5kdc[8941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:46:20 dist-test-slave-vdtf krb5kdc[8941](info): set up 2 sockets
Sep 25 22:46:20 dist-test-slave-vdtf krb5kdc[8941](info): commencing operation
krb5kdc: starting...
W20260925 22:46:21.756348 806 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:46:21 dist-test-slave-vdtf krb5kdc[8941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:46:21.879017 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:44935
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:44935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:21.986989 8954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:21.987234 8954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:21.987272 8954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:21.991113 8954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:21.991164 8954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:21.991182 8954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:21.991199 8954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:21.991214 8954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:21.996400 8954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:44935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:44935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:21.997375 8954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:21.998310 8954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:46:22.006776 8954 server_base.cc:1034] running on GCE node
W20260925 22:46:22.006729 8959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:22.006744 8960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:22.006714 8962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:22.007586 8954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:22.008652 8954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:22.009838 8954 hybrid_clock.cc:648] HybridClock initialized: now 1790376382009830 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T22:46:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1299 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:46:22 dist-test-slave-vdtf krb5kdc[8967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:46:22 dist-test-slave-vdtf krb5kdc[8967](info): set up 2 sockets
Sep 25 22:46:22 dist-test-slave-vdtf krb5kdc[8967](info): commencing operation
krb5kdc: starting...
W20260925 22:46:23.048341 806 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:46:23 dist-test-slave-vdtf krb5kdc[8967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:46:23.167308 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:45025
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:23.276717 8980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:23.277005 8980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:23.277087 8980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:23.281093 8980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:23.281168 8980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:23.281207 8980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:23.281265 8980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:23.281304 8980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:23.286365 8980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:45025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:23.287328 8980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:23.288352 8980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:23.296423 8988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:23.296450 8985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:23.296602 8980 server_base.cc:1034] running on GCE node
W20260925 22:46:23.296732 8986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:23.297317 8980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:23.298396 8980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:23.299583 8980 hybrid_clock.cc:648] HybridClock initialized: now 1790376383299566 us; error 32 us; skew 500 ppm
Sep 25 22:46:23 dist-test-slave-vdtf krb5kdc[8967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:23.306595 8980 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:46:23.307458 8980 webserver.cc:460] Webserver started at http://127.0.201.190:35947/ using document root <none> and password file <none>
I20260925 22:46:23.308082 8980 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:23.308193 8980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:23.308432 8980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:23.310477 8980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "59efffb8354d45938b3d4ea2e95a0042"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "d2143dfea762a6da3e0387e26173b50a"
server_key_iv: "105aea09d2c91aaa2680bc975f811c87"
server_key_version: "encryptionkey@0"
I20260925 22:46:23.311034 8980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "59efffb8354d45938b3d4ea2e95a0042"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "d2143dfea762a6da3e0387e26173b50a"
server_key_iv: "105aea09d2c91aaa2680bc975f811c87"
server_key_version: "encryptionkey@0"
I20260925 22:46:23.314764 8980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:46:23.317297 8995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:23.318508 8980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:46:23.318647 8980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "59efffb8354d45938b3d4ea2e95a0042"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "d2143dfea762a6da3e0387e26173b50a"
server_key_iv: "105aea09d2c91aaa2680bc975f811c87"
server_key_version: "encryptionkey@0"
I20260925 22:46:23.318758 8980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:23.338408 8980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:23.341533 8980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:23.341758 8980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:23.348763 8980 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:45025
I20260925 22:46:23.348784 9047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:45025 every 8 connection(s)
I20260925 22:46:23.349745 8980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:23.352509 9048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:23.353673 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 8980
I20260925 22:46:23.353780 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:23.354089 806 external_mini_cluster.cc:1428] Setting key f83e17d48d488cf01429adc84b599f20
I20260925 22:46:23.358278 9048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042: Bootstrap starting.
I20260925 22:46:23.360487 9048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:23.361337 9048 log.cc:824] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:46:23 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:23.363579 9048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042: No bootstrap required, opened a new log
I20260925 22:46:23.367730 9048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } }
I20260925 22:46:23.367957 9048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:23.368021 9048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59efffb8354d45938b3d4ea2e95a0042, State: Initialized, Role: FOLLOWER
I20260925 22:46:23.368036 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:23.355488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.0.201.190:45025)
0925 22:46:23.355906 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:23.355920 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:23.355955 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:46:23.356569 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:23.356584 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:23.356915 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:23.357254 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:23.358060 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:23.359057 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:23.359986 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:23.360450 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:23.360620 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:23.363475 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:23.363479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:23.363491 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:23.363516 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:23.365199 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:23.365732 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:23.365736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:23.365740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:23.365840 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:23.366121 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:23.366123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:23.366124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:23.366475 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:23.366626 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:23.366899 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 22:46:23.367185 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
I20260925 22:46:23.368491 9048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } }
I20260925 22:46:23.368619 9048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:23.368729 9048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:23.368872 9048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:23.369966 9048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } }
I20260925 22:46:23.370312 9048 leader_election.cc:302] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59efffb8354d45938b3d4ea2e95a0042; no voters:
I20260925 22:46:23.370657 9048 leader_election.cc:288] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:23.370771 9053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:23.371007 9053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 1 LEADER]: Becoming Leader. State: Replica: 59efffb8354d45938b3d4ea2e95a0042, State: Running, Role: LEADER
I20260925 22:46:23.371342 9053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } }
I20260925 22:46:23.371930 9048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:23.373140 9054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59efffb8354d45938b3d4ea2e95a0042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } } }
I20260925 22:46:23.373185 9055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59efffb8354d45938b3d4ea2e95a0042. Latest consensus state: current_term: 1 leader_uuid: "59efffb8354d45938b3d4ea2e95a0042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } } }
I20260925 22:46:23.373265 9054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:23.373265 9055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:23.373800 9062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:23.376434 9062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:23.381327 9062 catalog_manager.cc:1347] Generated new cluster ID: 0ec5b68cbd484fb4a8931d44296068b5
I20260925 22:46:23.381388 9062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:23.396930 9062 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:23.398133 9062 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:23.409861 9062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042: Generated new TSK 0
I20260925 22:46:23.410795 9062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:46:23.449111 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:45025
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:23.558041 9074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:23.558296 9074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:23.558339 9074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:23.562235 9074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:23.562285 9074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:23.562341 9074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:23.567338 9074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:23.568382 9074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:23.569327 9074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:23.578807 9082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:23.578845 9079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:23.578841 9080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:23.578989 9074 server_base.cc:1034] running on GCE node
I20260925 22:46:23.579474 9074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:23.580154 9074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:23.581344 9074 hybrid_clock.cc:648] HybridClock initialized: now 1790376383581334 us; error 30 us; skew 500 ppm
Sep 25 22:46:23 dist-test-slave-vdtf krb5kdc[8967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:23.583983 9074 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:46:23.584802 9074 webserver.cc:460] Webserver started at http://127.0.201.129:38567/ using document root <none> and password file <none>
I20260925 22:46:23.585350 9074 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:23.585438 9074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:23.585695 9074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:23.587730 9074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "1d146ffa4eb544d2b72c60b51eb79dde"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "585e8622e7cf59698aac4ea9aee6d469"
server_key_iv: "7dcb04c8b930b3f670be329ae7727ed0"
server_key_version: "encryptionkey@0"
I20260925 22:46:23.588310 9074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "1d146ffa4eb544d2b72c60b51eb79dde"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "585e8622e7cf59698aac4ea9aee6d469"
server_key_iv: "7dcb04c8b930b3f670be329ae7727ed0"
server_key_version: "encryptionkey@0"
I20260925 22:46:23.591971 9074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:46:23.594424 9089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:23.595726 9074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:23.595861 9074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "1d146ffa4eb544d2b72c60b51eb79dde"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "585e8622e7cf59698aac4ea9aee6d469"
server_key_iv: "7dcb04c8b930b3f670be329ae7727ed0"
server_key_version: "encryptionkey@0"
I20260925 22:46:23.595964 9074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:23.613168 9074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:23.616441 9074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:23.616652 9074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:23.617228 9074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:23.618036 9074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:23.618110 9074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:23.618181 9074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:23.618225 9074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:23.627317 9074 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:32975
I20260925 22:46:23.627338 9202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:32975 every 8 connection(s)
I20260925 22:46:23.628206 9074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:23.635548 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 9074
I20260925 22:46:23.635679 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:23.636052 806 external_mini_cluster.cc:1428] Setting key 7274ac08cde57343a086648384ccfe43
Sep 25 22:46:23 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:23.642408 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:23.630066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:59157 (local address 127.0.201.190:45025)
0925 22:46:23.630238 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:23.630242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:23.631097 (+ 855us) server_negotiation.cc:407] Connection header received
0925 22:46:23.632481 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:23.632484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:23.632529 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:23.632632 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:23.633754 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:23.634270 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:23.635412 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:23.635758 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:23.635850 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:23.638553 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:23.638557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:23.638564 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:23.638590 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:23.640176 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:23.640821 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:23.640824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:23.640825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:23.640895 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:23.641290 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:23.641292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:23.641293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:23.641450 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:23.641520 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:23.642190 (+ 670us) server_negotiation.cc:299] Negotiation successful
0925 22:46:23.642289 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:46:23.643774 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:23.630490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:59157)
0925 22:46:23.630943 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:23.630985 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:23.632285 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:23.632779 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:23.632786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:23.633144 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:23.633626 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:23.633639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:23.634374 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:23.634377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:23.635295 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:23.635305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:23.635905 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:23.635910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:23.636107 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:23.636735 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:23.636751 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:23.638372 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:23.640294 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:23.640303 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:23.640322 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:23.640720 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:23.640987 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:23.640993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:23.640998 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:23.641199 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:23.641558 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:23.641565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:23.641913 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 22:46:23.642156 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 22:46:23.642533 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260925 22:46:23.645047 9203 heartbeater.cc:347] Connected to a master server at 127.0.201.190:45025
I20260925 22:46:23.645288 9203 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:23.645780 9203 heartbeater.cc:511] Master 127.0.201.190:45025 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:46:23.647291 9012 ts_manager.cc:194] Registered new tserver with Master: 1d146ffa4eb544d2b72c60b51eb79dde (127.0.201.129:32975)
I20260925 22:46:23.648427 9012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:59157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:46:23.664943 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:45025
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:23.772218 9208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:23.772462 9208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:23.772552 9208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:23.776623 9208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:23.776700 9208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:23.776787 9208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:23.781883 9208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:23.782867 9208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:23.783908 9208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:23.793000 9213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:23.793090 9208 server_base.cc:1034] running on GCE node
W20260925 22:46:23.793008 9214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:23.793015 9216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:23.793692 9208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:23.794251 9208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:23.795413 9208 hybrid_clock.cc:648] HybridClock initialized: now 1790376383795396 us; error 26 us; skew 500 ppm
Sep 25 22:46:23 dist-test-slave-vdtf krb5kdc[8967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:23.798372 9208 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:46:23.799271 9208 webserver.cc:460] Webserver started at http://127.0.201.130:39035/ using document root <none> and password file <none>
I20260925 22:46:23.799970 9208 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:23.800032 9208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:23.800256 9208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:23.802228 9208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "238e72427df348338b7780c9bf8e4e2c"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "fe8906ddd7946cf3d736089afb8ec10c"
server_key_iv: "f176970a977f212f57cb9ff9c6735faa"
server_key_version: "encryptionkey@0"
I20260925 22:46:23.802778 9208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "238e72427df348338b7780c9bf8e4e2c"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "fe8906ddd7946cf3d736089afb8ec10c"
server_key_iv: "f176970a977f212f57cb9ff9c6735faa"
server_key_version: "encryptionkey@0"
I20260925 22:46:23.806380 9208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:23.808943 9223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:23.810194 9208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:46:23.810325 9208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "238e72427df348338b7780c9bf8e4e2c"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "fe8906ddd7946cf3d736089afb8ec10c"
server_key_iv: "f176970a977f212f57cb9ff9c6735faa"
server_key_version: "encryptionkey@0"
I20260925 22:46:23.810437 9208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:23.829171 9208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:23.832140 9208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:23.832290 9208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:23.832782 9208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:23.833513 9208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:23.833557 9208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:23.833595 9208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:23.833612 9208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:23.844492 9208 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:45373
I20260925 22:46:23.844509 9336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:45373 every 8 connection(s)
I20260925 22:46:23.845366 9208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:23.850886 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 9208
I20260925 22:46:23.851033 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:23.851330 806 external_mini_cluster.cc:1428] Setting key d4a32cf7fdbe46d9fd1c22b0d1a4eb26
Sep 25 22:46:23 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:23.859602 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:23.847024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:35957 (local address 127.0.201.190:45025)
0925 22:46:23.847166 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:23.847170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:23.848059 (+ 889us) server_negotiation.cc:407] Connection header received
0925 22:46:23.849365 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:23.849368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:23.849410 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:23.849485 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:23.850648 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:23.851272 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:23.852822 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:23.853131 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:23.853242 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:23.856111 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:23.856115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:23.856118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:23.856142 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:23.857610 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:23.858129 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:23.858132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:23.858134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:23.858212 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:23.858544 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:23.858547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:23.858548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:23.858725 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:23.858831 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:23.859315 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 22:46:23.859430 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:46:23.860488 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:23.847466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:35957)
0925 22:46:23.847919 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:23.847960 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:23.849181 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:23.849663 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:23.849670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:23.850013 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:23.850506 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:23.850517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:23.851392 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:23.851397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:23.852664 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:23.852676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:23.853408 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:23.853415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:23.853646 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:23.854334 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:23.854359 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:23.855953 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:23.857725 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:23.857731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:23.857746 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:23.858014 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:23.858299 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:23.858303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:23.858305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:23.858442 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:23.858828 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:23.858834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:23.859079 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 22:46:23.859263 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 22:46:23.859583 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":103,"threads_started":1}
I20260925 22:46:23.861660 9337 heartbeater.cc:347] Connected to a master server at 127.0.201.190:45025
I20260925 22:46:23.861918 9337 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:23.862432 9337 heartbeater.cc:511] Master 127.0.201.190:45025 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
I20260925 22:46:23.863552 9012 ts_manager.cc:194] Registered new tserver with Master: 238e72427df348338b7780c9bf8e4e2c (127.0.201.130:45373)
I20260925 22:46:23.864166 9012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:35957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:46:23.878782 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:45025
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:23.989398 9342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:23.989650 9342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:23.989689 9342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:23.993724 9342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:23.993773 9342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:23.993866 9342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:23.999150 9342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:24.000335 9342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:24.001327 9342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:24.010422 9347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:24.010433 9350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:24.010569 9342 server_base.cc:1034] running on GCE node
W20260925 22:46:24.010633 9348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:24.011066 9342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:24.011759 9342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:24.012964 9342 hybrid_clock.cc:648] HybridClock initialized: now 1790376384012950 us; error 33 us; skew 500 ppm
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:24.015978 9342 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:46:24.016779 9342 webserver.cc:460] Webserver started at http://127.0.201.131:37025/ using document root <none> and password file <none>
I20260925 22:46:24.017320 9342 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:24.017379 9342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:24.017614 9342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:24.019531 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "9212032c5614413eb5aa6dad317c3435"
format_stamp: "Formatted at 2026-09-25 22:46:24 on dist-test-slave-vdtf"
server_key: "3335274d98a35c4fbde7f0064d784c77"
server_key_iv: "7afa73e29603832117552ef07feff2f2"
server_key_version: "encryptionkey@0"
I20260925 22:46:24.020133 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "9212032c5614413eb5aa6dad317c3435"
format_stamp: "Formatted at 2026-09-25 22:46:24 on dist-test-slave-vdtf"
server_key: "3335274d98a35c4fbde7f0064d784c77"
server_key_iv: "7afa73e29603832117552ef07feff2f2"
server_key_version: "encryptionkey@0"
I20260925 22:46:24.024058 9342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:46:24.026397 9357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.027812 9342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:24.027936 9342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "9212032c5614413eb5aa6dad317c3435"
format_stamp: "Formatted at 2026-09-25 22:46:24 on dist-test-slave-vdtf"
server_key: "3335274d98a35c4fbde7f0064d784c77"
server_key_iv: "7afa73e29603832117552ef07feff2f2"
server_key_version: "encryptionkey@0"
I20260925 22:46:24.028044 9342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:24.060469 9342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:24.063635 9342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:24.063917 9342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:24.064529 9342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:24.065317 9342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:24.065361 9342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.065400 9342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:24.065421 9342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.074272 9342 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:33681
I20260925 22:46:24.074337 9470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:33681 every 8 connection(s)
I20260925 22:46:24.075250 9342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:24.084930 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 9342
I20260925 22:46:24.085076 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:46:24.085450 806 external_mini_cluster.cc:1428] Setting key 191f0d67b289766597cdda2c6752665d
I20260925 22:46:24.089299 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.077126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:43005 (local address 127.0.201.190:45025)
0925 22:46:24.077322 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.077327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.078175 (+ 848us) server_negotiation.cc:407] Connection header received
0925 22:46:24.079500 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.079503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.079591 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.079669 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.080860 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.081379 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.082532 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.082748 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.082849 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.085318 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:24.085322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:24.085325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:24.085353 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:24.087209 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:24.087846 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:24.087848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:24.087850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:24.087923 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:24.088310 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:24.088313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:24.088314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:24.088494 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:24.088606 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:24.089080 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 22:46:24.089186 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 22:46:24.090188 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.077571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:43005)
0925 22:46:24.078014 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:24.078056 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:24.079288 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:24.079884 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:24.079891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:24.080242 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:24.080728 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.080740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.081508 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.081511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.082415 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.082424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.082876 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.082879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.083012 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.083559 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:24.083574 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:24.085144 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:24.087336 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:24.087340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:24.087352 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:24.087747 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:24.088042 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:24.088045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:24.088047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:24.088189 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:24.088612 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:24.088617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:24.088843 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 22:46:24.088999 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 22:46:24.089257 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260925 22:46:24.091295 9471 heartbeater.cc:347] Connected to a master server at 127.0.201.190:45025
I20260925 22:46:24.091521 9471 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:24.092065 9471 heartbeater.cc:511] Master 127.0.201.190:45025 requested a full tablet report, sending...
I20260925 22:46:24.093052 9012 ts_manager.cc:194] Registered new tserver with Master: 9212032c5614413eb5aa6dad317c3435 (127.0.201.131:33681)
I20260925 22:46:24.093605 9012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:43005
I20260925 22:46:24.099488 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab.
Entry for principal oryx/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab.
WARNING: no policy specified for oryx/127.0.201.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab.
Entry for principal oryx/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab.
WARNING: no policy specified for oryx/127.0.201.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab.
Entry for principal oryx/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab.
I20260925 22:46:24.178222 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 9074
I20260925 22:46:24.183832 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 9208
I20260925 22:46:24.189213 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 9342
I20260925 22:46:24.194430 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 8980
I20260925 22:46:24.200625 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:45025
--webserver_interface=127.0.201.190
--webserver_port=35947
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:24.309829 9480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:24.310091 9480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:24.310130 9480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:24.314215 9480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:24.314270 9480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:24.314288 9480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:24.314304 9480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:24.314320 9480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:24.319496 9480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:45025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=35947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:24.320580 9480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:24.321595 9480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:24.329721 9488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:24.329777 9485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:24.329721 9486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:24.330057 9480 server_base.cc:1034] running on GCE node
I20260925 22:46:24.330766 9480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:24.331869 9480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:24.333051 9480 hybrid_clock.cc:648] HybridClock initialized: now 1790376384333026 us; error 38 us; skew 500 ppm
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:24.335831 9480 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:46:24.336607 9480 webserver.cc:460] Webserver started at http://127.0.201.190:35947/ using document root <none> and password file <none>
I20260925 22:46:24.337112 9480 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:24.337196 9480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:24.340878 9480 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 22:46:24.342841 9495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.344108 9480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:24.344228 9480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "59efffb8354d45938b3d4ea2e95a0042"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "d2143dfea762a6da3e0387e26173b50a"
server_key_iv: "105aea09d2c91aaa2680bc975f811c87"
server_key_version: "encryptionkey@0"
I20260925 22:46:24.344744 9480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:24.357007 9480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:24.359719 9480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:24.359911 9480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:24.366719 9480 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:45025
I20260925 22:46:24.366802 9547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:45025 every 8 connection(s)
I20260925 22:46:24.367791 9480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:24.371979 9548 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:46:24.373584 9548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042: Bootstrap starting.
I20260925 22:46:24.376390 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 9480
I20260925 22:46:24.376940 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:32975
--local_ip_for_outbound_sockets=127.0.201.129
--tserver_master_addrs=127.0.201.190:45025
--webserver_port=38567
--webserver_interface=127.0.201.129
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.201.129
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:46:24.390666 9548 log.cc:824] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:24.397109 9548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:46:24.397490 9548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042: Bootstrap complete.
I20260925 22:46:24.401870 9548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } }
I20260925 22:46:24.402261 9548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59efffb8354d45938b3d4ea2e95a0042, State: Initialized, Role: FOLLOWER
I20260925 22:46:24.402746 9548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } }
I20260925 22:46:24.402864 9548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:24.402949 9548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:24.403052 9548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:46:24.404983 9548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } }
I20260925 22:46:24.405326 9548 leader_election.cc:302] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59efffb8354d45938b3d4ea2e95a0042; no voters:
I20260925 22:46:24.405661 9548 leader_election.cc:288] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:46:24.405746 9552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:46:24.405923 9552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [term 2 LEADER]: Becoming Leader. State: Replica: 59efffb8354d45938b3d4ea2e95a0042, State: Running, Role: LEADER
I20260925 22:46:24.406198 9552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } }
I20260925 22:46:24.406849 9548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:24.407094 9552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59efffb8354d45938b3d4ea2e95a0042. Latest consensus state: current_term: 2 leader_uuid: "59efffb8354d45938b3d4ea2e95a0042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } } }
I20260925 22:46:24.407202 9552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:24.407549 9556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:24.407579 9553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "59efffb8354d45938b3d4ea2e95a0042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59efffb8354d45938b3d4ea2e95a0042" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 45025 } } }
I20260925 22:46:24.407836 9553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:24.410903 9556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:24.411551 9556 catalog_manager.cc:1259] Loaded cluster ID: 0ec5b68cbd484fb4a8931d44296068b5
I20260925 22:46:24.411612 9556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:24.413949 9556 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:24.414600 9556 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 59efffb8354d45938b3d4ea2e95a0042: Loaded TSK: 0
I20260925 22:46:24.416715 9556 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:46:24.489249 9550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:24.489535 9550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:24.489617 9550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:24.493723 9550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:24.493813 9550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:24.493907 9550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:24.499228 9550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.129.keytab
--principal=oryx/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:32975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=38567
--tserver_master_addrs=127.0.201.190:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:24.500277 9550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:24.501282 9550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:24.511763 9575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:24.511731 9574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:24.511829 9550 server_base.cc:1034] running on GCE node
W20260925 22:46:24.512110 9577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:24.512562 9550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:24.513249 9550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:24.514422 9550 hybrid_clock.cc:648] HybridClock initialized: now 1790376384514401 us; error 37 us; skew 500 ppm
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:24.517212 9550 init.cc:377] Logged in from keytab as oryx/127.0.201.129@KRBTEST.COM (short username oryx)
I20260925 22:46:24.518148 9550 webserver.cc:460] Webserver started at http://127.0.201.129:38567/ using document root <none> and password file <none>
I20260925 22:46:24.518725 9550 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:24.518796 9550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:24.522753 9550 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:46:24.524879 9584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.526319 9550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:46:24.526407 9550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "1d146ffa4eb544d2b72c60b51eb79dde"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "585e8622e7cf59698aac4ea9aee6d469"
server_key_iv: "7dcb04c8b930b3f670be329ae7727ed0"
server_key_version: "encryptionkey@0"
I20260925 22:46:24.526958 9550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:24.544032 9550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:24.547099 9550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:24.547317 9550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:24.548019 9550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:24.548904 9550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:24.548950 9550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.548993 9550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:24.549009 9550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.557982 9550 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:32975
I20260925 22:46:24.558005 9697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:32975 every 8 connection(s)
I20260925 22:46:24.558959 9550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:24.563293 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 9550
I20260925 22:46:24.563853 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:45373
--local_ip_for_outbound_sockets=127.0.201.130
--tserver_master_addrs=127.0.201.190:45025
--webserver_port=39035
--webserver_interface=127.0.201.130
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.201.130
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:24.573024 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.561196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:33745)
0925 22:46:24.561704 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:24.561744 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:24.562982 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:24.564509 (+ 1527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:24.564515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:24.564850 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:24.565288 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.565301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.566903 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.566906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.567919 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.567929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.569060 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.569064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.569206 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.569827 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:24.569842 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:24.572264 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":131,"threads_started":1}
W20260925 22:46:24.573537 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:24.561058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:33745 (local address 127.0.201.190:45025)
0925 22:46:24.561567 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.561592 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.561857 (+ 265us) server_negotiation.cc:407] Connection header received
0925 22:46:24.563413 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.563428 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.563789 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.564137 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.565456 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.566758 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.568051 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.568881 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.569076 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.572573 (+ 3497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:33745: BlockingRecv error: recv got EOF from 127.0.201.129:33745 (error 108)
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":116,"threads_started":1}
W20260925 22:46:24.573653 9698 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:45025 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.201.190:45025: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:24.579792 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.574191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:59101)
0925 22:46:24.574359 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:24.574377 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:24.574463 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:24.574737 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:24.574739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:24.574901 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:24.575005 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.575011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.575793 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.575797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.576688 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.576698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.577175 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.577178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.577270 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.577801 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:24.577814 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:24.579646 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260925 22:46:24.580040 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:24.574315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:59101 (local address 127.0.201.190:45025)
0925 22:46:24.574421 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.574424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.574436 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:46:24.574588 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.574590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.574629 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.574699 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.575108 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.575626 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.576834 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.577068 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.577154 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.579927 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:59101: BlockingRecv error: recv got EOF from 127.0.201.129:59101 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:24.585361 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.580441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:40449)
0925 22:46:24.580562 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:24.580576 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:24.580652 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:24.580852 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:24.580854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:24.580972 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:24.581035 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.581039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.582059 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.582061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.582552 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.582559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.583088 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.583091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.583145 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.583635 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:24.583646 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:24.585275 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:46:24.585515 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:24.580539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:40449 (local address 127.0.201.190:45025)
0925 22:46:24.580648 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.580652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.580665 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:46:24.580708 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.580710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.580751 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.580836 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.581153 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.581950 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.582651 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.582982 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.583076 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.585412 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:40449: BlockingRecv error: recv got EOF from 127.0.201.129:40449 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 22:46:24.585563 9698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:46:24.678681 9702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:24.678953 9702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:24.678996 9702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:24.683082 9702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:24.683148 9702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:24.683207 9702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:24.688398 9702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.130.keytab
--principal=oryx/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:45373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=39035
--tserver_master_addrs=127.0.201.190:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:24.689383 9702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:24.690395 9702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:24.700031 9708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:24.700031 9707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:24.700042 9710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:24.700342 9702 server_base.cc:1034] running on GCE node
I20260925 22:46:24.700744 9702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:24.701392 9702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:24.702576 9702 hybrid_clock.cc:648] HybridClock initialized: now 1790376384702566 us; error 38 us; skew 500 ppm
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:24.705376 9702 init.cc:377] Logged in from keytab as oryx/127.0.201.130@KRBTEST.COM (short username oryx)
I20260925 22:46:24.706151 9702 webserver.cc:460] Webserver started at http://127.0.201.130:39035/ using document root <none> and password file <none>
I20260925 22:46:24.706797 9702 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:24.706897 9702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:24.710961 9702 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:24.713016 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.714282 9702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:46:24.714383 9702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "238e72427df348338b7780c9bf8e4e2c"
format_stamp: "Formatted at 2026-09-25 22:46:23 on dist-test-slave-vdtf"
server_key: "fe8906ddd7946cf3d736089afb8ec10c"
server_key_iv: "f176970a977f212f57cb9ff9c6735faa"
server_key_version: "encryptionkey@0"
I20260925 22:46:24.714897 9702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:24.729470 9702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:24.732882 9702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:24.733095 9702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:24.733665 9702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:24.734467 9702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:24.734514 9702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.734556 9702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:24.734617 9702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.743378 9702 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:45373
I20260925 22:46:24.743404 9830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:45373 every 8 connection(s)
I20260925 22:46:24.744361 9702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:24.750610 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 9702
I20260925 22:46:24.751060 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:33681
--local_ip_for_outbound_sockets=127.0.201.131
--tserver_master_addrs=127.0.201.190:45025
--webserver_port=37025
--webserver_interface=127.0.201.131
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.201.131
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:46:24.756173 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:24.746109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:58231 (local address 127.0.201.190:45025)
0925 22:46:24.746238 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.746242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.747105 (+ 863us) server_negotiation.cc:407] Connection header received
0925 22:46:24.748519 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.748523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.748593 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.748690 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.749771 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.750281 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.751579 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.751870 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.751978 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.756078 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:58231: BlockingRecv error: recv got EOF from 127.0.201.130:58231 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:46:24.757031 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.746469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:58231)
0925 22:46:24.746952 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:24.746992 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:24.748304 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:24.748854 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:24.748861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:24.749210 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:24.749638 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.749650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.750394 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.750397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.751436 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.751445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.752011 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.752015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.752259 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.752854 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:24.752878 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:24.755969 (+ 3091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":102,"threads_started":1}
W20260925 22:46:24.757489 9831 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:45025 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.201.190:45025: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:24.762635 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.757928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:46319)
0925 22:46:24.758015 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:24.758031 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:24.758106 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:24.758343 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:24.758346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:24.758495 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:24.758574 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.758579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.759278 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.759284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.759870 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.759878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.760302 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.760306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.760391 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.760865 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:24.760876 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:24.762519 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 22:46:24.762769 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:24.758010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:46319 (local address 127.0.201.190:45025)
0925 22:46:24.758138 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.758144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.758159 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:46:24.758197 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.758199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.758248 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.758331 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.758688 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.759170 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.759979 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.760192 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.760303 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.762633 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:46319: BlockingRecv error: recv got EOF from 127.0.201.130:46319 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:24.768216 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.763174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:40325)
0925 22:46:24.763272 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:24.763287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:24.763352 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:24.763647 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:24.763649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:24.763831 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:24.763898 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.763903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.764939 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.764946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.765481 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.765489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.766011 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.766014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.766066 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.766523 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:24.766534 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:24.768135 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:46:24.768369 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:24.763234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:40325 (local address 127.0.201.190:45025)
0925 22:46:24.763339 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.763343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.763356 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:46:24.763412 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.763417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.763455 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.763555 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.763998 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.764774 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.765584 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.765916 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.766008 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.768267 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:40325: BlockingRecv error: recv got EOF from 127.0.201.130:40325 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 22:46:24.768571 9831 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:46:24.862370 9834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:24.862633 9834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:24.862676 9834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:24.866633 9834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:24.866688 9834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:24.866743 9834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:24.871825 9834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/krb5kdc/oryx_127.0.201.131.keytab
--principal=oryx/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:33681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=37025
--tserver_master_addrs=127.0.201.190:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:24.872762 9834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:24.873687 9834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:24.883180 9839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:24.883226 9840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:24.883183 9842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:24.883509 9834 server_base.cc:1034] running on GCE node
I20260925 22:46:24.883992 9834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:24.884624 9834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:24.885797 9834 hybrid_clock.cc:648] HybridClock initialized: now 1790376384885781 us; error 26 us; skew 500 ppm
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376384, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:24.888676 9834 init.cc:377] Logged in from keytab as oryx/127.0.201.131@KRBTEST.COM (short username oryx)
I20260925 22:46:24.889566 9834 webserver.cc:460] Webserver started at http://127.0.201.131:37025/ using document root <none> and password file <none>
I20260925 22:46:24.890079 9834 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:24.890148 9834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:24.894109 9834 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:46:24.896363 9849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.897688 9834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:46:24.897807 9834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "9212032c5614413eb5aa6dad317c3435"
format_stamp: "Formatted at 2026-09-25 22:46:24 on dist-test-slave-vdtf"
server_key: "3335274d98a35c4fbde7f0064d784c77"
server_key_iv: "7afa73e29603832117552ef07feff2f2"
server_key_version: "encryptionkey@0"
I20260925 22:46:24.898342 9834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:24.913179 9834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:24.916455 9834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:24.916661 9834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:24.917264 9834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:24.918231 9834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:24.918300 9834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.918367 9834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:24.918416 9834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:24.927181 9834 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:33681
I20260925 22:46:24.927243 9962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:33681 every 8 connection(s)
I20260925 22:46:24.928288 9834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:24.937736 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 9834
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:46:24.939851 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:24.930020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:51405 (local address 127.0.201.190:45025)
0925 22:46:24.930201 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.930206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.931074 (+ 868us) server_negotiation.cc:407] Connection header received
0925 22:46:24.932592 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.932595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.932641 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.932721 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.933862 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.934390 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.935502 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.935794 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.935874 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.939742 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:51405: BlockingRecv error: recv got EOF from 127.0.201.131:51405 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:46:24.940532 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.930496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:51405)
0925 22:46:24.930914 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:24.930957 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:24.932381 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:24.932884 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:24.932891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:24.933228 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:24.933715 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.933729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.934499 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.934502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.935391 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.935400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.935918 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.935921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.936060 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.936625 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:24.936642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:24.939621 (+ 2979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
W20260925 22:46:24.940999 9963 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:45025 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.201.190:45025: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:24.946735 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.941465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:47593)
0925 22:46:24.941572 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:24.941588 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:24.941663 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:24.941932 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:24.941935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:24.942098 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:24.942208 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.942214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.943151 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.943153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.943774 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.943785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.944240 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.944244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.944346 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.944807 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:24.944818 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:24.946597 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:46:24.946813 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:24.941529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:47593 (local address 127.0.201.190:45025)
0925 22:46:24.941638 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.941642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.941655 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:46:24.941738 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.941741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.941780 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.941859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.942320 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.943042 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.943909 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.944113 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.944240 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.946700 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:47593: BlockingRecv error: recv got EOF from 127.0.201.131:47593 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:46:24.948594 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.938736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.0.201.190:45025)
0925 22:46:24.938951 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.938962 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.938974 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:46:24.939002 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.939003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.939038 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.939117 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.940054 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.940649 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.941924 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.942302 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.942391 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.943846 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:24.943855 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:24.943879 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:24.943921 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:24.946013 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:24.946526 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:24.946533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:24.946544 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:24.946668 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:24.947177 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:24.947180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:24.947181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:24.947715 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:24.947953 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:24.948185 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 22:46:24.948436 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:24 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:24.952819 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:24.947388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:44247)
0925 22:46:24.947492 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:24.947507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:24.947610 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:24.947864 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:24.947867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:24.948067 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:24.948153 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.948160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.949032 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.949035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.949773 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:24.949781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.950312 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.950315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:24.950414 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.951021 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:24.951032 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:24.952737 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:46:24.952956 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:24.947448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:44247 (local address 127.0.201.190:45025)
0925 22:46:24.947567 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:24.947571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:24.947584 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:46:24.947715 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:24.947719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:24.947761 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:24.947835 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:24.948259 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.948907 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.949883 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:24.950189 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:24.950289 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:24.952847 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:44247: BlockingRecv error: recv got EOF from 127.0.201.131:44247 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 22:46:24.953052 9963 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 22:46:25 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:25 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:25.591995 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:25.586166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:55021)
0925 22:46:25.586485 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:25.586500 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:25.586594 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:25.586816 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:25.586818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:25.587003 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:25.587096 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:25.587102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.588020 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.588025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:25.588773 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:25.588783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.589374 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.589376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:25.589439 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:25.590020 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:25.590032 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:25.591876 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":85,"threads_started":1}
W20260925 22:46:25.592216 9969 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:25.586293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:55021 (local address 127.0.201.190:45025)
0925 22:46:25.586530 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:25.586533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:25.586548 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:46:25.586685 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:25.586687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:25.586731 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:25.586807 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:25.587206 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.587853 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.588931 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.589256 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.589365 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:25.592020 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:55021: BlockingRecv error: recv got EOF from 127.0.201.129:55021 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
Sep 25 22:46:25 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:25 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:25.774521 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:25.769283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:35781)
0925 22:46:25.769539 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:25.769553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:25.769655 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:25.769880 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:25.769882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:25.770051 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:25.770157 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:25.770162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.770853 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.770855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:25.771419 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:25.771426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.771881 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.771884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:25.771951 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:25.772423 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:25.772435 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:25.774354 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
W20260925 22:46:25.774629 9969 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:25.769339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:35781 (local address 127.0.201.190:45025)
0925 22:46:25.769469 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:25.769472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:25.769618 (+ 146us) server_negotiation.cc:407] Connection header received
0925 22:46:25.769736 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:25.769738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:25.769779 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:25.769838 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:25.770248 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.770760 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.771513 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.771791 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.771867 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:25.774503 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:35781: BlockingRecv error: recv got EOF from 127.0.201.130:35781 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 22:46:25 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:25 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:25.959151 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:25.953681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:49051)
0925 22:46:25.953965 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:25.953979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:25.954076 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:25.954318 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:25.954320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:25.954492 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:25.954585 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:25.954590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.955352 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.955355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:25.956024 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:25.956034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.956507 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.956510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:25.956581 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:25.957112 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:25.957124 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:25.958994 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":115,"threads_started":1}
W20260925 22:46:25.959317 9969 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:25.953792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:49051 (local address 127.0.201.190:45025)
0925 22:46:25.953929 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:25.953932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:25.954069 (+ 137us) server_negotiation.cc:407] Connection header received
0925 22:46:25.954170 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:25.954172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:25.954219 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:25.954283 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:25.954693 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.955243 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.956159 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:25.956379 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:25.956467 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:25.959158 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:49051: BlockingRecv error: recv got EOF from 127.0.201.131:49051 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 22:46:26 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:26 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:26.598300 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:26.592801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:44619)
0925 22:46:26.593068 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:26.593081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:26.593176 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:26.593422 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:26.593424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:26.593592 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:26.593735 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:26.593742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.594676 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.594679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:26.595353 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:26.595361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.595836 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.595838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:26.595898 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:26.596457 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:26.596469 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:26.598194 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
W20260925 22:46:26.598496 9973 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:26.592896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:44619 (local address 127.0.201.190:45025)
0925 22:46:26.593148 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:26.593152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:26.593166 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:46:26.593268 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:26.593270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:26.593313 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:26.593389 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:26.593847 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.594567 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.595479 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.595738 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.595830 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:26.598327 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:44619: BlockingRecv error: recv got EOF from 127.0.201.129:44619 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":56,"threads_started":1}
Sep 25 22:46:26 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:26 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:26.780815 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:26.775256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:58075)
0925 22:46:26.775572 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:26.775589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:26.775753 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:26.776014 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:26.776016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:26.776220 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:26.776306 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:26.776311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.777049 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.777055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:26.777790 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:26.777798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.778282 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.778289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:26.778410 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:26.778888 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:26.778898 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:26.780684 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":82,"threads_started":1}
W20260925 22:46:26.780948 9973 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:26.775391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:58075 (local address 127.0.201.190:45025)
0925 22:46:26.775540 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:26.775544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:26.775667 (+ 123us) server_negotiation.cc:407] Connection header received
0925 22:46:26.775853 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:26.775857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:26.775909 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:26.775996 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:26.776419 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.776940 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.777897 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.778149 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.778263 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:26.780808 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:58075: BlockingRecv error: recv got EOF from 127.0.201.130:58075 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:46:26 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:26 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:26.965659 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:26.960044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:59573)
0925 22:46:26.960498 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:26.960515 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:26.960706 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:26.961015 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:26.961018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:26.961221 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:26.961310 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:26.961315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.962063 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.962068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:26.962644 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:26.962652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.963096 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.963100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:26.963183 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:26.963764 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:26.963776 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:26.965534 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":128,"threads_started":1}
W20260925 22:46:26.965834 9973 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:26.960252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:59573 (local address 127.0.201.190:45025)
0925 22:46:26.960497 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:26.960503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:26.960619 (+ 116us) server_negotiation.cc:407] Connection header received
0925 22:46:26.960833 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:26.960836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:26.960903 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:26.960993 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:26.961429 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.961950 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.962763 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:26.962985 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:26.963095 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:26.965671 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:59573: BlockingRecv error: recv got EOF from 127.0.201.131:59573 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 25 22:46:27 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:27 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:27.604625 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:27.599046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:48401)
0925 22:46:27.599350 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:27.599364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:27.599461 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:27.599718 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:27.599721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:27.599898 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:27.599994 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:27.600000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.600735 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.600738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:27.601291 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:27.601298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.601849 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.601852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:27.601915 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:27.602436 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:27.602446 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:27.604438 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":70,"threads_started":1}
W20260925 22:46:27.604763 9977 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:27.599103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:48401 (local address 127.0.201.190:45025)
0925 22:46:27.599363 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:27.599366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:27.599413 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:46:27.599525 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:27.599527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:27.599569 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:27.599639 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:27.600094 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.600629 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.601442 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.601743 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.601830 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:27.604582 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:48401: BlockingRecv error: recv got EOF from 127.0.201.129:48401 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":127,"threads_started":1}
Sep 25 22:46:27 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:27 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:27.787065 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:27.781545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:41655)
0925 22:46:27.781846 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:27.781863 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:27.782006 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:27.782282 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:27.782285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:27.782483 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:27.782570 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:27.782575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.783358 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.783364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:27.783977 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:27.783984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.784443 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.784446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:27.784529 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:27.785018 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:27.785029 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:27.786869 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":80,"threads_started":1}
W20260925 22:46:27.787102 9977 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:27.781666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:41655 (local address 127.0.201.190:45025)
0925 22:46:27.781827 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:27.781832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:27.781948 (+ 116us) server_negotiation.cc:407] Connection header received
0925 22:46:27.782103 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:27.782119 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:27.782170 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:27.782260 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:27.782724 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.783252 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.784093 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.784333 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.784443 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:27.787005 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:41655: BlockingRecv error: recv got EOF from 127.0.201.130:41655 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 22:46:27 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:27 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:27.971652 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:27.966409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:50281)
0925 22:46:27.966676 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:27.966690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:27.966784 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:27.966968 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:27.966969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:27.967150 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:27.967242 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:27.967246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.968049 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.968053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:27.968601 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:27.968609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.969073 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.969076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:27.969145 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:27.969709 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:27.969723 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:27.971519 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":111,"threads_started":1}
W20260925 22:46:27.971794 9977 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:27.966464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:50281 (local address 127.0.201.190:45025)
0925 22:46:27.966610 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:27.966614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:27.966761 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:46:27.966848 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:27.966850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:27.966894 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:27.966958 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:27.967342 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.967935 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.968743 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:27.968967 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:27.969046 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:27.971652 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:50281: BlockingRecv error: recv got EOF from 127.0.201.131:50281 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 22:46:28 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:28 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:28.615856 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:28.605489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:42603)
0925 22:46:28.605804 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:28.605822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:28.605944 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:28.606184 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:28.606186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:28.606381 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:28.606473 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:28.606478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.607728 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.607735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:28.608396 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:28.608405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.609184 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.609187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:28.609252 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:28.613599 (+ 4347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:28.613611 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:28.615726 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":65,"threads_started":1}
W20260925 22:46:28.616032 9981 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:28.605710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:42603 (local address 127.0.201.190:45025)
0925 22:46:28.605978 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:28.605981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:28.605994 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:46:28.606026 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:28.606028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:28.606068 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:28.606144 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:28.606622 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.607544 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.608512 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.609068 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.609189 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:28.615881 (+ 6692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:42603: BlockingRecv error: recv got EOF from 127.0.201.129:42603 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
Sep 25 22:46:28 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:28 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:28.793493 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:28.787832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:41077)
0925 22:46:28.788249 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:28.788266 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:28.788412 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:28.788705 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:28.788708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:28.788911 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:28.788998 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:28.789003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.789782 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.789789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:28.790338 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:28.790345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.790793 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.790796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:28.790879 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:28.791363 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:28.791375 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:28.793329 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":87,"threads_started":1}
W20260925 22:46:28.793561 9981 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:28.787957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:41077 (local address 127.0.201.190:45025)
0925 22:46:28.788235 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:28.788240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:28.788353 (+ 113us) server_negotiation.cc:407] Connection header received
0925 22:46:28.788510 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:28.788514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:28.788566 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:28.788655 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:28.789118 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.789673 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.790456 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.790684 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.790791 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:28.793455 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:41077: BlockingRecv error: recv got EOF from 127.0.201.130:41077 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
Sep 25 22:46:28 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:28 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:28.978235 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:28.972490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:56423)
0925 22:46:28.972796 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:28.972818 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:28.972928 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:28.973176 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:28.973178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:28.973376 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:28.973487 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:28.973493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.974312 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.974315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:28.974960 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:28.974969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.975429 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.975431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:28.975501 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:28.976189 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:28.976202 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:28.978109 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":101,"threads_started":1}
W20260925 22:46:28.978453 9981 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:28.972526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:56423 (local address 127.0.201.190:45025)
0925 22:46:28.972709 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:28.972716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:28.972926 (+ 210us) server_negotiation.cc:407] Connection header received
0925 22:46:28.973001 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:28.973004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:28.973052 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:28.973131 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:28.973605 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.974199 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.975083 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:28.975320 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:28.975408 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:28.978271 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:56423: BlockingRecv error: recv got EOF from 127.0.201.131:56423 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 22:46:29 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:29 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:29.626689 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:29.616665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:48407)
0925 22:46:29.616956 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:29.616970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:29.617053 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:29.617322 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:29.617326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:29.617578 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:29.617675 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:29.617681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.618536 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.618538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:29.619138 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:29.619146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.619612 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.619615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:29.619753 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:29.624644 (+ 4891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:29.624660 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:29.626570 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":123,"threads_started":1}
W20260925 22:46:29.626905 9985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:29.616778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:48407 (local address 127.0.201.190:45025)
0925 22:46:29.617002 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:29.617005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:29.617020 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:46:29.617138 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:29.617140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:29.617189 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:29.617261 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:29.617803 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.618423 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.619234 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.619514 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.619589 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:29.626721 (+ 7132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:48407: BlockingRecv error: recv got EOF from 127.0.201.129:48407 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
Sep 25 22:46:29 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:29 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:29.801668 9986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:29.794379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:37751)
0925 22:46:29.794674 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:29.794689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:29.794817 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:29.795086 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:29.795091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:29.795349 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:29.795493 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:29.795501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.796315 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.796319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:29.796903 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:29.796911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.797593 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.797601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:29.797713 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:29.798686 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:29.798698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:29.801454 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
W20260925 22:46:29.802074 9985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:29.794518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:37751 (local address 127.0.201.190:45025)
0925 22:46:29.794664 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:29.794670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:29.794804 (+ 134us) server_negotiation.cc:407] Connection header received
0925 22:46:29.794892 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:29.794895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:29.794952 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:29.795029 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:29.795617 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.796170 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.797181 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.797410 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.797543 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:29.801673 (+ 4130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:37751: BlockingRecv error: recv got EOF from 127.0.201.130:37751 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 22:46:29 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:29 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:29.984781 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:29.979176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:59923)
0925 22:46:29.979575 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:29.979592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:29.979725 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:29.980026 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:29.980029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:29.980243 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:29.980337 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:29.980342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.981108 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.981113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:29.981701 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:29.981709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.982166 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.982170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:29.982258 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:29.982807 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:29.982825 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:29.984667 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":112,"threads_started":1}
W20260925 22:46:29.984889 9985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:29.979240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:59923 (local address 127.0.201.190:45025)
0925 22:46:29.979377 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:29.979382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:29.979664 (+ 282us) server_negotiation.cc:407] Connection header received
0925 22:46:29.979851 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:29.979854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:29.979918 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:29.980003 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:29.980451 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.980995 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.981828 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:29.982052 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:29.982164 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:29.984801 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:59923: BlockingRecv error: recv got EOF from 127.0.201.131:59923 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:46:30 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:30 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:30.633116 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:30.627518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:56223)
0925 22:46:30.627832 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:30.627845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:30.627944 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:30.628217 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:30.628220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:30.628402 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:30.628518 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:30.628524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.629260 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.629263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:30.629876 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:30.629884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.630405 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.630408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:30.630470 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:30.631073 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:30.631087 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:30.633006 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":84,"threads_started":1}
W20260925 22:46:30.633334 9989 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:30.627565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:56223 (local address 127.0.201.190:45025)
0925 22:46:30.627845 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:30.627849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:30.627896 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:46:30.628035 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:30.628037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:30.628082 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:30.628157 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:30.628625 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.629159 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.629994 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.630289 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.630370 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:30.633173 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:56223: BlockingRecv error: recv got EOF from 127.0.201.129:56223 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":60,"threads_started":1}
Sep 25 22:46:30 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:30 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:30.808060 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:30.802802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:41431)
0925 22:46:30.803116 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:30.803129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:30.803271 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:30.803490 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:30.803492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:30.803728 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:30.803820 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:30.803825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.804528 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.804532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:30.805087 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:30.805095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.805491 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.805494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:30.805552 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:30.806071 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:30.806081 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:30.807949 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":99,"threads_started":1}
W20260925 22:46:30.808229 9989 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:30.802914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:41431 (local address 127.0.201.190:45025)
0925 22:46:30.803057 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:30.803061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:30.803232 (+ 171us) server_negotiation.cc:407] Connection header received
0925 22:46:30.803342 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:30.803344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:30.803387 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:30.803458 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:30.803896 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.804408 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.805188 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.805398 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.805471 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:30.808077 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:41431: BlockingRecv error: recv got EOF from 127.0.201.130:41431 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 22:46:30 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:30 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:30.991029 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:30.985711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:55779)
0925 22:46:30.985952 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:30.985967 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:30.986074 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:30.986340 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:30.986342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:30.986535 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:30.986629 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:30.986634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.987433 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.987436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:30.988040 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:30.988049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.988490 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.988493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:30.988554 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:30.989122 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:30.989137 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:30.990915 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
W20260925 22:46:30.991261 9989 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:30.985763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:55779 (local address 127.0.201.190:45025)
0925 22:46:30.985889 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:30.985894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:30.986054 (+ 160us) server_negotiation.cc:407] Connection header received
0925 22:46:30.986138 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:30.986140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:30.986199 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:30.986270 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:30.986779 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.987318 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.988170 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:30.988392 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:30.988474 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:30.991117 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:55779: BlockingRecv error: recv got EOF from 127.0.201.131:55779 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 22:46:31 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:31 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:31.639544 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:31.633887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:57497)
0925 22:46:31.634229 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:31.634243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:31.634373 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:31.634672 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:31.634675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:31.634909 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:31.635023 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:31.635032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.636020 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.636022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:31.636604 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:31.636612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.637055 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.637058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:31.637118 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:31.637614 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:31.637626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:31.639432 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":68,"threads_started":1}
W20260925 22:46:31.639719 9993 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:31.634042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:57497 (local address 127.0.201.190:45025)
0925 22:46:31.634303 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:31.634308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:31.634326 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:46:31.634487 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:31.634490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:31.634536 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:31.634648 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:31.635165 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.635921 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.636724 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.636955 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.637034 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:31.639560 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:57497: BlockingRecv error: recv got EOF from 127.0.201.129:57497 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
Sep 25 22:46:31 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:31 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:31.813887 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:31.808814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:50517)
0925 22:46:31.809159 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:31.809173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:31.809279 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:31.809504 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:31.809506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:31.809679 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:31.809771 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:31.809776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.810483 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.810485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:31.811017 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:31.811023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.811420 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.811423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:31.811480 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:31.811960 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:31.811971 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:31.813777 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
W20260925 22:46:31.814138 9993 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:31.808866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:50517 (local address 127.0.201.190:45025)
0925 22:46:31.809056 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:31.809062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:31.809240 (+ 178us) server_negotiation.cc:407] Connection header received
0925 22:46:31.809356 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:31.809359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:31.809401 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:31.809472 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:31.809873 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.810392 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.811112 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.811328 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.811404 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:31.814022 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:50517: BlockingRecv error: recv got EOF from 127.0.201.130:50517 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:46:31 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:31 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:31.997081 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:31.991907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:45523)
0925 22:46:31.992256 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:31.992270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:31.992362 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:31.992570 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:31.992572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:31.992770 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:31.992859 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:31.992864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.993560 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.993562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:31.994126 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:31.994133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.994540 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.994543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:31.994603 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:31.995121 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:31.995132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:31.996952 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":92,"threads_started":1}
W20260925 22:46:31.997274 9993 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:31.991957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:45523 (local address 127.0.201.190:45025)
0925 22:46:31.992101 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:31.992105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:31.992322 (+ 217us) server_negotiation.cc:407] Connection header received
0925 22:46:31.992424 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:31.992426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:31.992470 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:31.992534 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:31.992953 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.993473 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.994221 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:31.994439 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:31.994513 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:31.997102 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:45523: BlockingRecv error: recv got EOF from 127.0.201.131:45523 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 22:46:32 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:32 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:32.646293 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:32.640416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:51411)
0925 22:46:32.640744 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:32.640767 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:32.640864 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:32.641166 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:32.641168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:32.641391 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:32.641516 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:32.641527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:32.642413 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:32.642420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:32.643044 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:32.643053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:32.643549 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:32.643552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:32.643644 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:32.644218 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:32.644240 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:32.646118 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":126,"threads_started":1}
W20260925 22:46:32.646370 9997 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:32.640631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:51411 (local address 127.0.201.190:45025)
0925 22:46:32.640890 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:32.640894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:32.640915 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:46:32.640952 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:32.640954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:32.640995 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:32.641137 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:32.641657 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:32.642292 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:32.643174 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:32.643430 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:32.643549 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:32.646240 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:51411: BlockingRecv error: recv got EOF from 127.0.201.129:51411 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
Sep 25 22:46:32 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:32 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:32.821223 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:32.814882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:54377)
0925 22:46:32.815218 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:32.815239 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:32.815374 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:32.815827 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:32.815833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:32.816083 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:32.816207 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:32.816218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:32.817249 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:32.817252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:32.817825 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:32.817833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:32.818286 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:32.818290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:32.818379 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:32.818923 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:32.818935 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:32.821041 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
W20260925 22:46:32.821281 9997 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:32.814940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:54377 (local address 127.0.201.190:45025)
0925 22:46:32.815158 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:32.815163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:32.815321 (+ 158us) server_negotiation.cc:407] Connection header received
0925 22:46:32.815518 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:32.815523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:32.815605 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:32.815712 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:32.816349 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:32.817148 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:32.817932 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:32.818160 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:32.818271 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:32.821168 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:54377: BlockingRecv error: recv got EOF from 127.0.201.130:54377 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 25 22:46:32 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:32 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:33.003721 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:32.997889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:50245)
0925 22:46:32.998181 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:32.998198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:32.998296 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:32.998527 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:32.998529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:32.998729 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:32.998825 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:32.998830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:32.999568 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:32.999572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:33.000230 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:33.000241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:33.000730 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.000733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:33.000822 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:33.001302 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:33.001313 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:33.003533 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":124,"threads_started":1}
W20260925 22:46:33.003770 9997 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:32.998104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:50245 (local address 127.0.201.190:45025)
0925 22:46:32.998244 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:32.998247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:32.998270 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:46:32.998380 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:32.998382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:32.998428 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:32.998489 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:32.998931 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:32.999446 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.000338 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:33.000609 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.000725 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:33.003656 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:50245: BlockingRecv error: recv got EOF from 127.0.201.131:50245 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 22:46:33 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:33 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:33.658878 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:33.647160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:59551)
0925 22:46:33.647433 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:33.647447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:33.647542 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:33.647782 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:33.647784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:33.647958 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:33.648052 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:33.648058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:33.648926 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.648929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:33.649472 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:33.649479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:33.649916 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.649918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:33.649976 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:33.656521 (+ 6545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:33.656536 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:33.658719 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
W20260925 22:46:33.659011 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:33.647204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:59551 (local address 127.0.201.190:45025)
0925 22:46:33.647456 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:33.647459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:33.647498 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:46:33.647606 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:33.647608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:33.647655 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:33.647747 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:33.648198 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:33.648829 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.649569 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:33.649812 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.649896 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:33.658880 (+ 8984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:59551: BlockingRecv error: recv got EOF from 127.0.201.129:59551 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
Sep 25 22:46:33 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:33 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:33.827152 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:33.822011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:59369)
0925 22:46:33.822268 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:33.822282 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:33.822376 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:33.822598 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:33.822600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:33.822798 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:33.822889 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:33.822894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:33.823586 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.823588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:33.824206 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:33.824215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:33.824641 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.824643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:33.824703 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:33.825204 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:33.825215 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:33.827040 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":91,"threads_started":1}
W20260925 22:46:33.827329 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:33.822053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:59369 (local address 127.0.201.190:45025)
0925 22:46:33.822202 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:33.822205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:33.822348 (+ 143us) server_negotiation.cc:407] Connection header received
0925 22:46:33.822454 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:33.822456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:33.822498 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:33.822565 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:33.822982 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:33.823502 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.824317 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:33.824540 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:33.824620 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:33.827195 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:59369: BlockingRecv error: recv got EOF from 127.0.201.130:59369 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 22:46:34 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:34 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:34.009991 10003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:34.004500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:40519)
0925 22:46:34.004771 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:34.004784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:34.004878 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:34.005099 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:34.005101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:34.005277 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:34.005367 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:34.005372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.006078 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.006081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:34.006757 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:34.006766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.007167 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.007190 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:34.007284 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:34.007833 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:34.007845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:34.009843 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
W20260925 22:46:34.010106 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:34.004537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:40519 (local address 127.0.201.190:45025)
0925 22:46:34.004668 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:34.004676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:34.004848 (+ 172us) server_negotiation.cc:407] Connection header received
0925 22:46:34.004953 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:34.004955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:34.005000 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:34.005066 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:34.005465 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.005979 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.006868 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.007077 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.007154 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:34.009992 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:40519: BlockingRecv error: recv got EOF from 127.0.201.131:40519 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 22:46:34 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:34 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:34.665036 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:34.659803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:42939)
0925 22:46:34.660060 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:34.660073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:34.660164 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:34.660389 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:34.660391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:34.660583 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:34.660675 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:34.660680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.661383 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.661386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:34.661958 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:34.661966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.662403 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.662406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:34.662472 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:34.662938 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:34.662950 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:34.664884 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
W20260925 22:46:34.665215 10005 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:34.659848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:42939 (local address 127.0.201.190:45025)
0925 22:46:34.660074 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:34.660078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:34.660139 (+ 61us) server_negotiation.cc:407] Connection header received
0925 22:46:34.660240 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:34.660242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:34.660283 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:34.660359 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:34.660768 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.661285 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.662062 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.662278 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.662355 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:34.665031 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:42939: BlockingRecv error: recv got EOF from 127.0.201.129:42939 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
Sep 25 22:46:34 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:34 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:34.833730 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:34.828057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:41269)
0925 22:46:34.828340 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:34.828355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:34.828489 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:34.828763 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:34.828765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:34.828977 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:34.829072 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:34.829077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.829896 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.829900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:34.830780 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:34.830792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.831304 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.831307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:34.831370 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:34.831911 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:34.831922 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:34.833621 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
W20260925 22:46:34.833863 10005 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:34.828111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:41269 (local address 127.0.201.190:45025)
0925 22:46:34.828285 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:34.828290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:34.828437 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:46:34.828602 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:34.828605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:34.828657 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:34.828748 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:34.829186 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.829766 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.830881 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:34.831195 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:34.831277 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:34.833749 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:41269: BlockingRecv error: recv got EOF from 127.0.201.130:41269 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 22:46:35 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:35 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:35.018486 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:35.011910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:44591)
0925 22:46:35.012196 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:35.012219 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:35.012316 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:35.012575 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:35.012577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:35.012765 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:35.012860 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:35.012865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.013898 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.013905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:35.014943 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:35.014958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.015650 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.015654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:35.015779 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:35.016400 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:35.016415 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:35.018338 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":120,"threads_started":1}
W20260925 22:46:35.018627 10005 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:35.012052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:44591 (local address 127.0.201.190:45025)
0925 22:46:35.012187 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:35.012190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:35.012290 (+ 100us) server_negotiation.cc:407] Connection header received
0925 22:46:35.012397 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:35.012399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:35.012461 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:35.012535 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:35.012970 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.013713 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.015154 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.015532 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.015619 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:35.018510 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:44591: BlockingRecv error: recv got EOF from 127.0.201.131:44591 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 22:46:35 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:35 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:35.671357 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:35.665798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:49669)
0925 22:46:35.666112 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:35.666126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:35.666242 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:35.666464 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:35.666466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:35.666653 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:35.666747 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:35.666752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.667539 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.667542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:35.668144 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:35.668154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.668551 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.668553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:35.668614 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:35.669266 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:35.669286 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:35.671184 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":84,"threads_started":1}
W20260925 22:46:35.671430 10009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:35.665920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:49669 (local address 127.0.201.190:45025)
0925 22:46:35.666145 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:35.666149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:35.666194 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:46:35.666308 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:35.666310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:35.666351 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:35.666423 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:35.666859 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.667418 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.668245 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.668456 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.668528 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:35.671309 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:49669: BlockingRecv error: recv got EOF from 127.0.201.129:49669 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
Sep 25 22:46:35 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:35 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:35.841738 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:35.834482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:33171)
0925 22:46:35.834780 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:35.834795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:35.834940 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:35.835244 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:35.835247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:35.835487 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:35.835615 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:35.835624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.836885 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.836892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:35.837943 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:35.837955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.838668 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.838673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:35.838787 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:35.839446 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:35.839460 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:35.841568 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":98,"threads_started":1}
W20260925 22:46:35.841889 10009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:35.834545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:33171 (local address 127.0.201.190:45025)
0925 22:46:35.834726 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:35.834731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:35.834879 (+ 148us) server_negotiation.cc:407] Connection header received
0925 22:46:35.835062 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:35.835064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:35.835111 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:35.835204 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:35.835760 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.836759 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.838065 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:35.838525 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:35.838673 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:35.841768 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:33171: BlockingRecv error: recv got EOF from 127.0.201.130:33171 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 22:46:36 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:36 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:36.024603 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:36.019285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:53693)
0925 22:46:36.019578 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:36.019594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:36.019756 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:36.020043 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:36.020045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:36.020212 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:36.020304 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:36.020309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.021020 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.021022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:36.021573 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:36.021580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.022035 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.022042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:36.022146 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:36.022622 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:36.022634 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:36.024494 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
W20260925 22:46:36.024780 10009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:36.019316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:53693 (local address 127.0.201.190:45025)
0925 22:46:36.019443 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:36.019447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:36.019720 (+ 273us) server_negotiation.cc:407] Connection header received
0925 22:46:36.019870 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:36.019873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:36.019938 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:36.020016 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:36.020408 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.020923 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.021691 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.021908 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.021987 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:36.024623 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:53693: BlockingRecv error: recv got EOF from 127.0.201.131:53693 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 22:46:36 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:36 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:36.677456 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:36.672138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:40233)
0925 22:46:36.672423 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:36.672436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:36.672535 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:36.672768 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:36.672770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:36.672942 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:36.673049 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:36.673055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.673807 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.673810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:36.674378 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:36.674387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.674786 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.674788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:36.674885 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:36.675476 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:36.675514 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:36.677342 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
W20260925 22:46:36.677613 10013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:36.672201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:40233 (local address 127.0.201.190:45025)
0925 22:46:36.672456 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:36.672459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:36.672488 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:46:36.672616 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:36.672618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:36.672660 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:36.672735 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:36.673149 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.673713 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.674478 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.674694 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.674769 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:36.677482 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:40233: BlockingRecv error: recv got EOF from 127.0.201.129:40233 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":63,"threads_started":1}
Sep 25 22:46:36 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:36 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:36.848382 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:36.842536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:55613)
0925 22:46:36.842862 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:36.842880 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:36.843041 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:36.843319 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:36.843322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:36.843530 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:36.843619 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:36.843625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.844450 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.844457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:36.845026 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:36.845033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.845495 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.845499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:36.845583 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:36.846109 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:36.846120 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:36.848207 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":85,"threads_started":1}
W20260925 22:46:36.848654 10013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:36.842672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:55613 (local address 127.0.201.190:45025)
0925 22:46:36.842846 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:36.842851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:36.842977 (+ 126us) server_negotiation.cc:407] Connection header received
0925 22:46:36.843153 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:36.843156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:36.843209 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:36.843297 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:36.843790 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.844325 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.845148 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:36.845377 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:36.845493 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:36.848379 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:55613: BlockingRecv error: recv got EOF from 127.0.201.130:55613 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 22:46:37 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:37 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:37.030869 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:37.025310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:45799)
0925 22:46:37.025574 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:37.025596 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:37.025719 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:37.025999 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:37.026002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:37.026219 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:37.026311 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:37.026316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.027084 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.027091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:37.027874 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:37.027885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.028361 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.028365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:37.028444 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:37.028922 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:37.028934 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:37.030751 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
W20260925 22:46:37.031131 10013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:37.025378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:45799 (local address 127.0.201.190:45025)
0925 22:46:37.025531 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:37.025536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:37.025663 (+ 127us) server_negotiation.cc:407] Connection header received
0925 22:46:37.025837 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:37.025840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:37.025897 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:37.025981 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:37.026420 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.026975 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.028007 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.028250 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.028359 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:37.030881 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:45799: BlockingRecv error: recv got EOF from 127.0.201.131:45799 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 22:46:37 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:37 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:37.683363 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:37.678237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:39453)
0925 22:46:37.678501 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:37.678525 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:37.678611 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:37.678831 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:37.678833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:37.679006 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:37.679095 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:37.679099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.679914 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.679917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:37.680470 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:37.680477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.680909 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.680911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:37.680968 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:37.681423 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:37.681434 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:37.683253 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
W20260925 22:46:37.683565 10017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:37.678364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:39453 (local address 127.0.201.190:45025)
0925 22:46:37.678621 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:37.678625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:37.678658 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:46:37.678689 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:37.678691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:37.678731 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:37.678797 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:37.679198 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.679823 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.680574 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.680815 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.680894 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:37.683405 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:39453: BlockingRecv error: recv got EOF from 127.0.201.129:39453 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
Sep 25 22:46:37 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:37 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:37.854457 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:37.849255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:52843)
0925 22:46:37.849532 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:37.849548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:37.849686 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:37.849940 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:37.849943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:37.850143 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:37.850237 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:37.850242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.850986 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.850993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:37.851548 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:37.851555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.852039 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.852042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:37.852120 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:37.852594 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:37.852605 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:37.854280 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
W20260925 22:46:37.854516 10017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:37.849380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:52843 (local address 127.0.201.190:45025)
0925 22:46:37.849525 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:37.849529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:37.849625 (+ 96us) server_negotiation.cc:407] Connection header received
0925 22:46:37.849784 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:37.849787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:37.849841 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:37.849924 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:37.850346 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.850880 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.851662 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:37.851927 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:37.852037 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:37.854403 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:52843: BlockingRecv error: recv got EOF from 127.0.201.130:52843 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 22:46:38 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:38 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:38.036829 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:38.031633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:38497)
0925 22:46:38.031950 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:38.031962 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:38.032065 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:38.032285 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:38.032287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:38.032454 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:38.032549 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:38.032554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.033244 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.033246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:38.033803 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:38.033812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.034246 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.034249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:38.034326 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:38.034834 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:38.034845 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:38.036661 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
W20260925 22:46:38.036895 10017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:38.031694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:38497 (local address 127.0.201.190:45025)
0925 22:46:38.031813 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:38.031818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:38.032040 (+ 222us) server_negotiation.cc:407] Connection header received
0925 22:46:38.032127 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:38.032129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:38.032177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:38.032245 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:38.032641 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.033149 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.033904 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.034122 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.034201 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:38.036773 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:38497: BlockingRecv error: recv got EOF from 127.0.201.131:38497 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 22:46:38 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:38 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:38.695151 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:38.684220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:42753)
0925 22:46:38.684569 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:38.684585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:38.684721 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:38.685088 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:38.685091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:38.685369 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:38.685505 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:38.685513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.686675 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.686682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:38.687673 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:38.687725 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.688511 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.688517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:38.688641 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:38.693171 (+ 4530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:38.693183 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:38.695026 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":74,"threads_started":1}
W20260925 22:46:38.695381 10021 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:38.684270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:42753 (local address 127.0.201.190:45025)
0925 22:46:38.684600 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:38.684606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:38.684651 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:46:38.684848 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:38.684862 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:38.684928 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:38.685042 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:38.685650 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.686545 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.687851 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.688244 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.688363 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:38.695230 (+ 6867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:42753: BlockingRecv error: recv got EOF from 127.0.201.129:42753 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":64,"threads_started":1}
Sep 25 22:46:38 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:38 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:38.860518 10022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:38.855191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:57751)
0925 22:46:38.855465 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:38.855479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:38.855577 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:38.855845 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:38.855848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:38.856070 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:38.856170 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:38.856175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.856930 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.856932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:38.857511 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:38.857518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.857965 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.857968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:38.858031 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:38.858538 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:38.858550 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:38.860408 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
W20260925 22:46:38.860651 10021 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:38.855330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:57751 (local address 127.0.201.190:45025)
0925 22:46:38.855500 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:38.855503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:38.855531 (+ 28us) server_negotiation.cc:407] Connection header received
0925 22:46:38.855643 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:38.855645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:38.855705 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:38.855775 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:38.856310 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.856833 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.857620 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:38.857849 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:38.857928 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:38.860536 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:57751: BlockingRecv error: recv got EOF from 127.0.201.130:57751 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 22:46:39 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:39 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.131@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:39.042953 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:39.037591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.131:35731)
0925 22:46:39.037863 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:39.037877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:39.037973 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:39.038212 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:39.038215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:39.038408 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:39.038516 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:39.038522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.039324 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.039327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:39.040066 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:39.040075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.040479 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.040482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:39.040540 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:39.041069 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:39.041081 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:39.042832 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":105,"threads_started":1}
W20260925 22:46:39.043111 10021 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:39.037634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:35731 (local address 127.0.201.190:45025)
0925 22:46:39.037766 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:39.037772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:39.037945 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:46:39.038037 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:39.038039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:39.038085 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:39.038151 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:39.038605 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.039139 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.040163 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.040385 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.040462 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:39.042954 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.131:35731: BlockingRecv error: recv got EOF from 127.0.201.131:35731 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 22:46:39 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:39 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.129@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:39.701646 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:39.696178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.129:44321)
0925 22:46:39.696453 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:39.696466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:39.696560 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:39.696891 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:39.696894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:39.697098 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:39.697194 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:39.697200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.697934 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.697936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:39.698511 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:39.698519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.698919 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.698921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:39.698979 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:39.699485 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:39.699496 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:39.701459 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
W20260925 22:46:39.701719 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:39.696231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:44321 (local address 127.0.201.190:45025)
0925 22:46:39.696477 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:39.696481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:39.696542 (+ 61us) server_negotiation.cc:407] Connection header received
0925 22:46:39.696682 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:39.696685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:39.696731 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:39.696817 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:39.697304 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.697831 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.698615 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.698826 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.698899 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:39.701577 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.129:44321: BlockingRecv error: recv got EOF from 127.0.201.129:44321 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
Sep 25 22:46:39 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:46:39 dist-test-slave-vdtf krb5kdc[8967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.201.130@KRBTEST.COM for oryx/127.0.201.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:46:39.868069 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:39.861492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:45025 (local address 127.0.201.130:37507)
0925 22:46:39.861770 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:39.861792 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:39.861910 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:39.862358 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:39.862360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:39.862603 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:39.862717 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:39.862726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.863923 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.863928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:39.864544 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:39.864553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.865139 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.865143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:39.865226 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:39.865846 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:39.865858 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:39.867886 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.201.190:45025: Server oryx/127.0.201.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":104,"threads_started":1}
W20260925 22:46:39.868321 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:39.861543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:37507 (local address 127.0.201.190:45025)
0925 22:46:39.861696 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:39.861700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:39.861883 (+ 183us) server_negotiation.cc:407] Connection header received
0925 22:46:39.862060 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:39.862066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:39.862127 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:39.862189 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:39.862928 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.863817 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.864669 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:39.865011 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:39.865132 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:39.868070 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.201.130:37507: BlockingRecv error: recv got EOF from 127.0.201.130:37507 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:46:39.947221 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 9550
I20260925 22:46:39.952987 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 9702
I20260925 22:46:39.958415 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 9834
I20260925 22:46:39.963385 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 9480
2026-09-25T22:46:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17947 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T22:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:39Z Disabled control of system clock
I20260925 22:46:39.993782 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:46301
--webserver_interface=127.0.201.190
--webserver_port=0
--builtin_ntp_servers=127.0.201.148:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:46301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:46:40.105715 10030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:40.105983 10030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:40.106017 10030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:40.110652 10030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:40.110728 10030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:40.110745 10030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:40.110762 10030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:40.110810 10030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:40.115546 10030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:46301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:40.116627 10030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:40.117964 10030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:40.126483 10035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:40.126673 10030 server_base.cc:1034] running on GCE node
W20260925 22:46:40.126531 10036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:40.126483 10038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:40.127329 10030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:40.128448 10030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:40.129659 10030 hybrid_clock.cc:648] HybridClock initialized: now 1790376400129628 us; error 45 us; skew 500 ppm
I20260925 22:46:40.131613 10030 webserver.cc:460] Webserver started at http://127.0.201.190:38995/ using document root <none> and password file <none>
I20260925 22:46:40.132259 10030 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:40.132325 10030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:40.132552 10030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:40.134630 10030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "2eb3003ada6f49bd93e471d875ca5f78"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "f37b55386b632412bebe03541a59a897"
server_key_iv: "8ea394930165f00ce5dc8f59f8dc0529"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.135205 10030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "2eb3003ada6f49bd93e471d875ca5f78"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "f37b55386b632412bebe03541a59a897"
server_key_iv: "8ea394930165f00ce5dc8f59f8dc0529"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.138944 10030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 22:46:40.141373 10044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:40.142613 10030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:46:40.142717 10030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "2eb3003ada6f49bd93e471d875ca5f78"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "f37b55386b632412bebe03541a59a897"
server_key_iv: "8ea394930165f00ce5dc8f59f8dc0529"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.142866 10030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:40.152859 10030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:40.153566 10030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:40.153764 10030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:40.160380 10030 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:46301
I20260925 22:46:40.160396 10096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:46301 every 8 connection(s)
I20260925 22:46:40.161370 10030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:40.164165 10097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:40.169354 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 10030
I20260925 22:46:40.169512 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:40.169716 10097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78: Bootstrap starting.
I20260925 22:46:40.169837 806 external_mini_cluster.cc:1428] Setting key d9517f1241490e389494297e307382bd
I20260925 22:46:40.172335 10097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:40.173066 10097 log.cc:824] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:40.174865 10097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78: No bootstrap required, opened a new log
I20260925 22:46:40.178309 10097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb3003ada6f49bd93e471d875ca5f78" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46301 } }
I20260925 22:46:40.178551 10097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:40.178630 10097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb3003ada6f49bd93e471d875ca5f78, State: Initialized, Role: FOLLOWER
I20260925 22:46:40.179021 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:40.171228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.0.201.190:46301)
0925 22:46:40.171941 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:40.171957 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:40.171993 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:46:40.172670 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:40.172688 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:40.173086 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:40.173465 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:40.173880 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.174821 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.175729 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.176193 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.176377 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:40.177225 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:40.177230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:40.177242 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:40.177617 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:40.177675 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:40.177693 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:40.177801 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:40.177938 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 22:46:40.178190 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":132,"threads_started":1}
I20260925 22:46:40.179147 10097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb3003ada6f49bd93e471d875ca5f78" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46301 } }
I20260925 22:46:40.179335 10097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:40.179406 10097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:40.179499 10097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:40.180467 10097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb3003ada6f49bd93e471d875ca5f78" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46301 } }
I20260925 22:46:40.180810 10097 leader_election.cc:302] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eb3003ada6f49bd93e471d875ca5f78; no voters:
I20260925 22:46:40.181136 10097 leader_election.cc:288] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:40.181267 10102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:40.181541 10102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [term 1 LEADER]: Becoming Leader. State: Replica: 2eb3003ada6f49bd93e471d875ca5f78, State: Running, Role: LEADER
I20260925 22:46:40.181849 10102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb3003ada6f49bd93e471d875ca5f78" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46301 } }
I20260925 22:46:40.182296 10097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:40.183496 10104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eb3003ada6f49bd93e471d875ca5f78. Latest consensus state: current_term: 1 leader_uuid: "2eb3003ada6f49bd93e471d875ca5f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb3003ada6f49bd93e471d875ca5f78" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46301 } } }
I20260925 22:46:40.183502 10103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eb3003ada6f49bd93e471d875ca5f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb3003ada6f49bd93e471d875ca5f78" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 46301 } } }
I20260925 22:46:40.183655 10104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:40.183706 10103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:40.184072 10112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:40.186833 10112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:40.192165 10112 catalog_manager.cc:1347] Generated new cluster ID: 016bf5253be742588ffff71a277d749a
I20260925 22:46:40.192230 10112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:40.201402 10112 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:40.202236 10112 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:40.206401 10112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eb3003ada6f49bd93e471d875ca5f78: Generated new TSK 0
I20260925 22:46:40.207024 10112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:46:40.218346 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46301
--builtin_ntp_servers=127.0.201.148:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 22:46:40.344916 10121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:40.345146 10121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:40.345181 10121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:40.349241 10121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:40.349287 10121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:40.349364 10121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:40.354004 10121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:40.354970 10121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:40.356389 10121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:40.365630 10127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:40.365605 10129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:40.365594 10126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:40.366029 10121 server_base.cc:1034] running on GCE node
I20260925 22:46:40.366413 10121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:40.367048 10121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:40.368208 10121 hybrid_clock.cc:648] HybridClock initialized: now 1790376400368196 us; error 24 us; skew 500 ppm
I20260925 22:46:40.370110 10121 webserver.cc:460] Webserver started at http://127.0.201.129:42263/ using document root <none> and password file <none>
I20260925 22:46:40.370676 10121 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:40.370739 10121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:40.370944 10121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:40.373046 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "63c5d08ae50d48fe95d6f04a406abde6"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "22b64d3c590c8e24055a06408c441e64"
server_key_iv: "043c2c8afd9cb687cd69b9539fe5f6b7"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.373620 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "63c5d08ae50d48fe95d6f04a406abde6"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "22b64d3c590c8e24055a06408c441e64"
server_key_iv: "043c2c8afd9cb687cd69b9539fe5f6b7"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.377738 10121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:46:40.380297 10135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:40.381603 10121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:40.381752 10121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "63c5d08ae50d48fe95d6f04a406abde6"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "22b64d3c590c8e24055a06408c441e64"
server_key_iv: "043c2c8afd9cb687cd69b9539fe5f6b7"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.381870 10121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:40.405102 10121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:40.405813 10121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:40.406018 10121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:40.406612 10121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:40.407541 10121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:40.407611 10121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:40.407698 10121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:40.407752 10121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:40.416343 10121 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:35693
I20260925 22:46:40.416419 10248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:35693 every 8 connection(s)
I20260925 22:46:40.417245 10121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:40.425150 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 10121
I20260925 22:46:40.425280 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:40.425688 806 external_mini_cluster.cc:1428] Setting key 089c67167326a40e2f702c6aa66e344e
I20260925 22:46:40.426446 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:40.419073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:44871 (local address 127.0.201.190:46301)
0925 22:46:40.419221 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:40.419224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:40.420292 (+ 1068us) server_negotiation.cc:407] Connection header received
0925 22:46:40.421640 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:40.421643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:40.421688 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:40.421766 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:40.422654 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.423177 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.424293 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.424522 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.424604 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:40.425954 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:40.425958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:40.425960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:40.425972 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:40.425992 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:40.426000 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:40.426060 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:40.426249 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 22:46:40.426315 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:46:40.427498 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:40.419454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:46301 (local address 127.0.201.129:44871)
0925 22:46:40.420145 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:40.420185 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:40.421444 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:40.421954 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:40.421961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:40.422041 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:40.422518 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:40.422530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.423285 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.423287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:40.424186 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:40.424194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.424684 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.424687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:40.424914 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:40.425644 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:40.425647 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:40.425701 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:40.425787 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:40.426091 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:40.426102 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:40.426108 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:46:40.426219 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:46:40.426510 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":110,"threads_started":1}
I20260925 22:46:40.427815 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46301
--builtin_ntp_servers=127.0.201.148:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:46:40.428803 10249 heartbeater.cc:347] Connected to a master server at 127.0.201.190:46301
I20260925 22:46:40.429046 10249 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:40.429714 10249 heartbeater.cc:511] Master 127.0.201.190:46301 requested a full tablet report, sending...
I20260925 22:46:40.431200 10061 ts_manager.cc:194] Registered new tserver with Master: 63c5d08ae50d48fe95d6f04a406abde6 (127.0.201.129:35693)
I20260925 22:46:40.432430 10061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.129:44871
W20260925 22:46:40.536839 10252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:40.537079 10252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:40.537147 10252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:40.541190 10252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:40.541245 10252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:40.541297 10252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:40.546101 10252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:40.547222 10252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:40.548625 10252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:40.558156 10258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:40.558240 10257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:40.558166 10260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:40.558655 10252 server_base.cc:1034] running on GCE node
I20260925 22:46:40.559121 10252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:40.559836 10252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:40.561023 10252 hybrid_clock.cc:648] HybridClock initialized: now 1790376400561007 us; error 30 us; skew 500 ppm
I20260925 22:46:40.562939 10252 webserver.cc:460] Webserver started at http://127.0.201.130:44545/ using document root <none> and password file <none>
I20260925 22:46:40.563510 10252 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:40.563602 10252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:40.563866 10252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:40.565913 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "3bf8fc00be0748bdb06a3681799f0540"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "9dd07088579a318ab44f9226f83749e1"
server_key_iv: "de61b05d27d6c9f3c28258037f3d5c67"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.566471 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "3bf8fc00be0748bdb06a3681799f0540"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "9dd07088579a318ab44f9226f83749e1"
server_key_iv: "de61b05d27d6c9f3c28258037f3d5c67"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.570181 10252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:40.572827 10266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:40.574096 10252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:46:40.574231 10252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "3bf8fc00be0748bdb06a3681799f0540"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "9dd07088579a318ab44f9226f83749e1"
server_key_iv: "de61b05d27d6c9f3c28258037f3d5c67"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.574347 10252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:40.596526 10252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:40.597297 10252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:40.597517 10252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:40.598141 10252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:40.599141 10252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:40.599217 10252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:40.599296 10252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:40.599340 10252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:40.607939 10252 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:33557
I20260925 22:46:40.607960 10379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:33557 every 8 connection(s)
I20260925 22:46:40.608831 10252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:40.614360 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 10252
I20260925 22:46:40.614485 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:40.614779 806 external_mini_cluster.cc:1428] Setting key b7fa5aa27db01ba09e65b80cd21d63cb
I20260925 22:46:40.616828 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46301
--builtin_ntp_servers=127.0.201.148:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:46:40.618659 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:40.610686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:38643 (local address 127.0.201.190:46301)
0925 22:46:40.610842 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:40.610847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:40.612004 (+ 1157us) server_negotiation.cc:407] Connection header received
0925 22:46:40.613375 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:40.613378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:40.613424 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:40.613500 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:40.614391 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.615086 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.616627 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.616834 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.616985 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:40.618155 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:40.618161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:40.618165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:40.618183 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:40.618209 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:40.618224 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:40.618310 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:40.618498 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 22:46:40.618549 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:46:40.620048 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:40.611130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:46301 (local address 127.0.201.130:38643)
0925 22:46:40.611857 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:40.611907 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:40.613166 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:40.613669 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:40.613675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:40.613759 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:40.614241 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:40.614253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.615233 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.615236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:40.616504 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:40.616515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.617022 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.617026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:40.617200 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:40.617849 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:40.617850 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:40.617894 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:40.617959 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:40.618319 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:40.618328 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:40.618337 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:46:40.618489 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 22:46:40.618926 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260925 22:46:40.621251 10380 heartbeater.cc:347] Connected to a master server at 127.0.201.190:46301
I20260925 22:46:40.621492 10380 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:40.622035 10380 heartbeater.cc:511] Master 127.0.201.190:46301 requested a full tablet report, sending...
I20260925 22:46:40.623114 10061 ts_manager.cc:194] Registered new tserver with Master: 3bf8fc00be0748bdb06a3681799f0540 (127.0.201.130:33557)
I20260925 22:46:40.623754 10061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.130:38643
W20260925 22:46:40.726413 10383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:40.726648 10383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:40.726681 10383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:40.730661 10383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:40.730706 10383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:40.730773 10383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:40.735539 10383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:40.736503 10383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:40.737784 10383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:40.746732 10388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:40.746659 10391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:40.746768 10383 server_base.cc:1034] running on GCE node
W20260925 22:46:40.746960 10389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:40.747359 10383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:40.748049 10383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:40.749203 10383 hybrid_clock.cc:648] HybridClock initialized: now 1790376400749189 us; error 26 us; skew 500 ppm
I20260925 22:46:40.750922 10383 webserver.cc:460] Webserver started at http://127.0.201.131:37889/ using document root <none> and password file <none>
I20260925 22:46:40.751477 10383 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:40.751535 10383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:40.751817 10383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:40.753784 10383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "a842f342fe1b47d3bc30fb2ef69c0dce"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "5ea50e17d1a9865ee201ea68043809fa"
server_key_iv: "c144e34db2d1724ddbe45551be9cf74c"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.754338 10383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "a842f342fe1b47d3bc30fb2ef69c0dce"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "5ea50e17d1a9865ee201ea68043809fa"
server_key_iv: "c144e34db2d1724ddbe45551be9cf74c"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.757993 10383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:46:40.760438 10397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:40.761641 10383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:46:40.761770 10383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "a842f342fe1b47d3bc30fb2ef69c0dce"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "5ea50e17d1a9865ee201ea68043809fa"
server_key_iv: "c144e34db2d1724ddbe45551be9cf74c"
server_key_version: "encryptionkey@0"
I20260925 22:46:40.761921 10383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:40.781243 10383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:40.781879 10383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:40.782061 10383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:40.782668 10383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:40.783546 10383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:40.783627 10383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:40.783689 10383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:40.783753 10383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:40.792451 10383 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:33349
I20260925 22:46:40.792474 10510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:33349 every 8 connection(s)
I20260925 22:46:40.793295 10383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:40.802217 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:40.795063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:44939 (local address 127.0.201.190:46301)
0925 22:46:40.795230 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:40.795234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:40.796300 (+ 1066us) server_negotiation.cc:407] Connection header received
0925 22:46:40.797662 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:40.797665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:40.797724 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:40.797835 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:40.798698 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.799225 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.800381 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.800596 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.800703 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:40.801741 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:40.801744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:40.801747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:40.801767 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:40.801785 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:40.801793 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:40.801875 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:40.802026 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:46:40.802082 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:46:40.803005 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:40.795466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:46301 (local address 127.0.201.131:44939)
0925 22:46:40.796151 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:40.796190 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:40.797449 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:40.798009 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:40.798015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:40.798096 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:40.798551 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:40.798563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.799361 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.799365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:40.800244 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:40.800253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:40.800735 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:40.800739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:40.800901 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:40.801495 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:40.801497 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:40.801533 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:40.801590 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:40.801887 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:40.801897 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:40.801909 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 22:46:40.802025 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:46:40.802279 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":105,"threads_started":1}
I20260925 22:46:40.803397 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 10383
I20260925 22:46:40.803520 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:46:40.803858 806 external_mini_cluster.cc:1428] Setting key 748f243dfb83ac74c82bc0422e1223d0
I20260925 22:46:40.804186 10511 heartbeater.cc:347] Connected to a master server at 127.0.201.190:46301
I20260925 22:46:40.804435 10511 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:40.805001 10511 heartbeater.cc:511] Master 127.0.201.190:46301 requested a full tablet report, sending...
I20260925 22:46:40.805920 10061 ts_manager.cc:194] Registered new tserver with Master: a842f342fe1b47d3bc30fb2ef69c0dce (127.0.201.131:33349)
I20260925 22:46:40.806458 10061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.131:44939
I20260925 22:46:40.817431 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:46:40.820411 10100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:40.819358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.0.201.190:46301)
0925 22:46:40.819524 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:40.819529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:40.819790 (+ 261us) server_negotiation.cc:407] Connection header received
0925 22:46:40.819899 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:40.819902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:40.819941 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:40.820007 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:40.820288 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35766: BlockingRecv error: recv got EOF from 127.0.0.1:35766 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 22:46:40.820529 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 10121
I20260925 22:46:40.826153 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 10252
I20260925 22:46:40.831252 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 10383
I20260925 22:46:40.836511 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 10030
2026-09-25T22:46:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (871 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T22:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:40Z Disabled control of system clock
I20260925 22:46:40.864930 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42019
--webserver_interface=127.0.201.190
--webserver_port=0
--builtin_ntp_servers=127.0.201.148:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:42019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:46:40.972616 10522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:40.972872 10522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:40.972909 10522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:40.976907 10522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:40.976967 10522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:40.976982 10522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:40.976999 10522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:40.977015 10522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:40.981801 10522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:42019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:40.982860 10522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:40.984278 10522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:40.992635 10528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:40.992640 10522 server_base.cc:1034] running on GCE node
W20260925 22:46:40.992581 10527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:40.992589 10530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:40.993348 10522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:40.994421 10522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:40.995607 10522 hybrid_clock.cc:648] HybridClock initialized: now 1790376400995596 us; error 31 us; skew 500 ppm
I20260925 22:46:40.997370 10522 webserver.cc:460] Webserver started at http://127.0.201.190:35909/ using document root <none> and password file <none>
I20260925 22:46:40.997927 10522 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:40.998018 10522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:40.998253 10522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:41.000347 10522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "5ba6e837589e4eddaa0f9ce599379e13"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "29a73839f303a29eb48cd3a55b032320"
server_key_iv: "4f4daffa89ff88bdb6ac90ad75b7816a"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.000890 10522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "5ba6e837589e4eddaa0f9ce599379e13"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "29a73839f303a29eb48cd3a55b032320"
server_key_iv: "4f4daffa89ff88bdb6ac90ad75b7816a"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.004465 10522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:46:41.006815 10536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:41.008044 10522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:41.008177 10522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "5ba6e837589e4eddaa0f9ce599379e13"
format_stamp: "Formatted at 2026-09-25 22:46:40 on dist-test-slave-vdtf"
server_key: "29a73839f303a29eb48cd3a55b032320"
server_key_iv: "4f4daffa89ff88bdb6ac90ad75b7816a"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.008296 10522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:41.012624 10522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:41.013182 10522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:41.013362 10522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:41.019526 10522 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:42019
I20260925 22:46:41.019542 10588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:42019 every 8 connection(s)
I20260925 22:46:41.020421 10522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:41.023486 10589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:41.028754 10589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13: Bootstrap starting.
I20260925 22:46:41.029951 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 10522
I20260925 22:46:41.030085 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:41.030365 806 external_mini_cluster.cc:1428] Setting key 038d1213d92988b49ea6f98f7129090a
I20260925 22:46:41.031394 10589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:41.032271 10589 log.cc:824] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:41.034505 10589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13: No bootstrap required, opened a new log
I20260925 22:46:41.036273 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:41.031523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.0.201.190:42019)
0925 22:46:41.032278 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:41.032298 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:41.032342 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:46:41.033170 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:41.033195 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:41.033547 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:41.033904 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:41.034699 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:41.034706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:41.034723 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:41.034737 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:41.034806 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:41.034830 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:41.034944 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:41.035073 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 22:46:41.035331 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":199,"threads_started":1}
I20260925 22:46:41.039326 10589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba6e837589e4eddaa0f9ce599379e13" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42019 } }
I20260925 22:46:41.039563 10589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:41.039621 10589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba6e837589e4eddaa0f9ce599379e13, State: Initialized, Role: FOLLOWER
I20260925 22:46:41.040026 10589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba6e837589e4eddaa0f9ce599379e13" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42019 } }
I20260925 22:46:41.040174 10589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:41.040239 10589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:41.040342 10589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:41.041175 10589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba6e837589e4eddaa0f9ce599379e13" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42019 } }
I20260925 22:46:41.041510 10589 leader_election.cc:302] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ba6e837589e4eddaa0f9ce599379e13; no voters:
I20260925 22:46:41.041806 10589 leader_election.cc:288] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:41.041918 10594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:41.042177 10594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [term 1 LEADER]: Becoming Leader. State: Replica: 5ba6e837589e4eddaa0f9ce599379e13, State: Running, Role: LEADER
I20260925 22:46:41.042459 10594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba6e837589e4eddaa0f9ce599379e13" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42019 } }
I20260925 22:46:41.042968 10589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:41.044487 10596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba6e837589e4eddaa0f9ce599379e13. Latest consensus state: current_term: 1 leader_uuid: "5ba6e837589e4eddaa0f9ce599379e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba6e837589e4eddaa0f9ce599379e13" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42019 } } }
I20260925 22:46:41.044607 10596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:41.044999 10595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ba6e837589e4eddaa0f9ce599379e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba6e837589e4eddaa0f9ce599379e13" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42019 } } }
I20260925 22:46:41.045090 10595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13 [sys.catalog]: This master's current role is: LEADER
W20260925 22:46:41.047407 10610 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:46:41.047494 10610 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:46:41.047569 10605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:41.048243 10605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:41.053083 10605 catalog_manager.cc:1347] Generated new cluster ID: c130ec44df71463eb70fd9b1fee47a53
I20260925 22:46:41.053145 10605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:41.081535 10605 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:41.081923 10605 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:41.087947 10605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ba6e837589e4eddaa0f9ce599379e13: Generated new TSK 0
I20260925 22:46:41.088822 10605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:46:41.093515 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42019
--builtin_ntp_servers=127.0.201.148:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:46:41.202641 10613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:41.202924 10613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:41.202997 10613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:41.206976 10613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:41.207120 10613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:41.207304 10613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:41.212601 10613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:41.213673 10613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:41.215051 10613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:41.224382 10618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:41.224452 10621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:41.224483 10619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:41.224824 10613 server_base.cc:1034] running on GCE node
I20260925 22:46:41.225252 10613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:41.225889 10613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:41.227046 10613 hybrid_clock.cc:648] HybridClock initialized: now 1790376401227016 us; error 42 us; skew 500 ppm
I20260925 22:46:41.228858 10613 webserver.cc:460] Webserver started at http://127.0.201.129:42741/ using document root <none> and password file <none>
I20260925 22:46:41.229452 10613 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:41.229511 10613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:41.229734 10613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:41.231813 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "8999a0be4e0d452a84a68c283c013f37"
format_stamp: "Formatted at 2026-09-25 22:46:41 on dist-test-slave-vdtf"
server_key: "1fe7ef66b7e365e3b5f233c43ce61d73"
server_key_iv: "a2e52fe8dc57513ec80e11c67b071e2c"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.232354 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "8999a0be4e0d452a84a68c283c013f37"
format_stamp: "Formatted at 2026-09-25 22:46:41 on dist-test-slave-vdtf"
server_key: "1fe7ef66b7e365e3b5f233c43ce61d73"
server_key_iv: "a2e52fe8dc57513ec80e11c67b071e2c"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.236019 10613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:46:41.238291 10627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:41.239493 10613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:41.239621 10613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "8999a0be4e0d452a84a68c283c013f37"
format_stamp: "Formatted at 2026-09-25 22:46:41 on dist-test-slave-vdtf"
server_key: "1fe7ef66b7e365e3b5f233c43ce61d73"
server_key_iv: "a2e52fe8dc57513ec80e11c67b071e2c"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.239773 10613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:41.244127 10613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:41.244777 10613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:41.244958 10613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:41.245527 10613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:41.246407 10613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:41.246450 10613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:41.246485 10613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:41.246507 10613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:41.254895 10613 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:36325
I20260925 22:46:41.254923 10740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:36325 every 8 connection(s)
I20260925 22:46:41.255803 10613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:41.259637 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 10613
I20260925 22:46:41.259778 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:41.260100 806 external_mini_cluster.cc:1428] Setting key 35cdc54c9dc94fc99fd819ee16cc3759
I20260925 22:46:41.261978 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:41.257596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:51907 (local address 127.0.201.190:42019)
0925 22:46:41.257733 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:41.257737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:41.258716 (+ 979us) server_negotiation.cc:407] Connection header received
0925 22:46:41.260125 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:41.260131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:41.260176 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:41.260274 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:41.261506 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:41.261509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:41.261512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:41.261524 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:41.261547 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:41.261557 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:41.261646 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:41.261781 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 22:46:41.261821 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:46:41.262112 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42019
--builtin_ntp_servers=127.0.201.148:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 22:46:41.262797 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:41.257968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42019 (local address 127.0.201.129:51907)
0925 22:46:41.258576 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:41.258630 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:41.259901 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:41.260415 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:41.260422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:41.260545 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:41.261188 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:41.261190 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:41.261234 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:41.261334 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:41.261650 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:41.261657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:41.261662 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:46:41.261792 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 22:46:41.261974 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260925 22:46:41.264231 10741 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42019
I20260925 22:46:41.264467 10741 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:41.264876 10741 heartbeater.cc:511] Master 127.0.201.190:42019 requested a full tablet report, sending...
I20260925 22:46:41.266372 10553 ts_manager.cc:194] Registered new tserver with Master: 8999a0be4e0d452a84a68c283c013f37 (127.0.201.129:36325)
W20260925 22:46:41.369485 10744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:41.369719 10744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:41.369755 10744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:41.373615 10744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:41.373657 10744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:41.373709 10744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:41.378182 10744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:41.379129 10744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:41.380407 10744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:41.389516 10749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:41.389456 10752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:41.389503 10750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:41.389775 10744 server_base.cc:1034] running on GCE node
I20260925 22:46:41.390113 10744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:41.390715 10744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:41.391863 10744 hybrid_clock.cc:648] HybridClock initialized: now 1790376401391848 us; error 26 us; skew 500 ppm
I20260925 22:46:41.393505 10744 webserver.cc:460] Webserver started at http://127.0.201.130:45027/ using document root <none> and password file <none>
I20260925 22:46:41.394066 10744 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:41.394127 10744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:41.394377 10744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:41.396382 10744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "dfc3d9ae23874f6687c4936d03eb0936"
format_stamp: "Formatted at 2026-09-25 22:46:41 on dist-test-slave-vdtf"
server_key: "0cf422ac271fd49228ad4778225faa3c"
server_key_iv: "9796b0135b0fb69b6e21c84c63740a25"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.396920 10744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "dfc3d9ae23874f6687c4936d03eb0936"
format_stamp: "Formatted at 2026-09-25 22:46:41 on dist-test-slave-vdtf"
server_key: "0cf422ac271fd49228ad4778225faa3c"
server_key_iv: "9796b0135b0fb69b6e21c84c63740a25"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.400771 10744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 22:46:41.403079 10758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:41.404336 10744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:41.404422 10744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "dfc3d9ae23874f6687c4936d03eb0936"
format_stamp: "Formatted at 2026-09-25 22:46:41 on dist-test-slave-vdtf"
server_key: "0cf422ac271fd49228ad4778225faa3c"
server_key_iv: "9796b0135b0fb69b6e21c84c63740a25"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.404552 10744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:41.409471 10744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:41.410023 10744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:41.410209 10744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:41.410712 10744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:41.411621 10744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:41.411666 10744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:41.411779 10744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:41.411818 10744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:41.420274 10744 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:44613
I20260925 22:46:41.420311 10871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:44613 every 8 connection(s)
I20260925 22:46:41.421067 10744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:41.426635 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:41.422712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:36723 (local address 127.0.201.190:42019)
0925 22:46:41.422840 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:41.422844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:41.423852 (+ 1008us) server_negotiation.cc:407] Connection header received
0925 22:46:41.425083 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:41.425085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:41.425120 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:41.425198 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:41.426274 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:41.426276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:41.426278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:41.426288 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:41.426304 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:41.426311 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:41.426382 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:41.426495 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 22:46:41.426530 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:46:41.427428 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:41.423078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42019 (local address 127.0.201.130:36723)
0925 22:46:41.423714 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:41.423772 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:41.424908 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:41.425351 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:41.425357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:41.425441 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:41.426032 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:41.426034 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:41.426070 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:41.426135 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:41.426386 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:41.426391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:41.426395 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:46:41.426518 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:46:41.426697 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260925 22:46:41.428556 10872 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42019
I20260925 22:46:41.428769 10872 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:41.429141 10872 heartbeater.cc:511] Master 127.0.201.190:42019 requested a full tablet report, sending...
I20260925 22:46:41.429358 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 10744
I20260925 22:46:41.429473 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:41.429838 806 external_mini_cluster.cc:1428] Setting key 26de08860d35feb802876d5208758016
I20260925 22:46:41.430266 10553 ts_manager.cc:194] Registered new tserver with Master: dfc3d9ae23874f6687c4936d03eb0936 (127.0.201.130:44613)
I20260925 22:46:41.431828 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42019
--builtin_ntp_servers=127.0.201.148:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:46:41.539870 10875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:41.540112 10875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:41.540184 10875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:41.544546 10875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:41.544610 10875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:41.544703 10875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:41.549548 10875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:41.550578 10875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:41.551997 10875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:41.562134 10881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:41.562153 10880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:41.562251 10875 server_base.cc:1034] running on GCE node
W20260925 22:46:41.562130 10883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:41.562805 10875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:41.563462 10875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:41.564627 10875 hybrid_clock.cc:648] HybridClock initialized: now 1790376401564608 us; error 68 us; skew 500 ppm
I20260925 22:46:41.566469 10875 webserver.cc:460] Webserver started at http://127.0.201.131:45877/ using document root <none> and password file <none>
I20260925 22:46:41.567143 10875 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:41.567205 10875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:41.567461 10875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:41.569584 10875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "68bd795d43dd4990a8089cc8f6e79c79"
format_stamp: "Formatted at 2026-09-25 22:46:41 on dist-test-slave-vdtf"
server_key: "2a178060b610255410603332559a6947"
server_key_iv: "6b9c334780711d2f67ee047477c22791"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.570160 10875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "68bd795d43dd4990a8089cc8f6e79c79"
format_stamp: "Formatted at 2026-09-25 22:46:41 on dist-test-slave-vdtf"
server_key: "2a178060b610255410603332559a6947"
server_key_iv: "6b9c334780711d2f67ee047477c22791"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.573870 10875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:41.576215 10889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:41.577479 10875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:41.577651 10875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "68bd795d43dd4990a8089cc8f6e79c79"
format_stamp: "Formatted at 2026-09-25 22:46:41 on dist-test-slave-vdtf"
server_key: "2a178060b610255410603332559a6947"
server_key_iv: "6b9c334780711d2f67ee047477c22791"
server_key_version: "encryptionkey@0"
I20260925 22:46:41.577765 10875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:41.582290 10875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:41.582870 10875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:41.583047 10875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:41.583566 10875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:41.584458 10875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:41.584529 10875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:41.584605 10875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:41.584659 10875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:41.593712 10875 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:42053
I20260925 22:46:41.593734 11002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:42053 every 8 connection(s)
I20260925 22:46:41.594589 10875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:41.597527 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 10875
I20260925 22:46:41.597631 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:46:41.597936 806 external_mini_cluster.cc:1428] Setting key 003daa4a9c3a0f7e3a4a19187fb0436d
I20260925 22:46:41.600917 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:41.596346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:57801 (local address 127.0.201.190:42019)
0925 22:46:41.596477 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:41.596480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:41.597550 (+ 1070us) server_negotiation.cc:407] Connection header received
0925 22:46:41.599329 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:41.599331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:41.599365 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:41.599432 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:41.600608 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:41.600611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:41.600613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:46:41.600622 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:41.600639 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:46:41.600647 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:41.600714 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:41.600792 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 22:46:41.600823 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:46:41.601779 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:41.596787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42019 (local address 127.0.201.131:57801)
0925 22:46:41.597403 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:41.597442 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:41.599107 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:41.599561 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:41.599568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:41.599710 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:41.600321 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:46:41.600324 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:41.600377 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:46:41.600464 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:41.600704 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:41.600709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:41.600712 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:46:41.600828 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:46:41.601032 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
I20260925 22:46:41.602774 11003 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42019
I20260925 22:46:41.602993 11003 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:41.603384 11003 heartbeater.cc:511] Master 127.0.201.190:42019 requested a full tablet report, sending...
I20260925 22:46:41.604396 10552 ts_manager.cc:194] Registered new tserver with Master: 68bd795d43dd4990a8089cc8f6e79c79 (127.0.201.131:42053)
I20260925 22:46:41.612136 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:46:41.615031 10592 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:46:41.614000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.0.201.190:42019)
0925 22:46:41.614152 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:41.614157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:41.614322 (+ 165us) server_negotiation.cc:407] Connection header received
0925 22:46:41.614505 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:41.614511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:41.614563 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:41.614649 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:41.614905 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49740: BlockingRecv error: recv got EOF from 127.0.0.1:49740 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:46:41.615375 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 10613
I20260925 22:46:41.621042 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 10744
I20260925 22:46:41.626351 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 10875
I20260925 22:46:41.631201 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 10522
2026-09-25T22:46:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (794 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:46:41 dist-test-slave-vdtf krb5kdc[11013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:46:41 dist-test-slave-vdtf krb5kdc[11013](info): set up 2 sockets
Sep 25 22:46:41 dist-test-slave-vdtf krb5kdc[11013](info): commencing operation
krb5kdc: starting...
W20260925 22:46:42.692793 806 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:46:42 dist-test-slave-vdtf krb5kdc[11013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:46:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:46:42.811379 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:43261
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:43261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:42.921583 11026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:42.921824 11026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:42.921864 11026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:42.925765 11026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:42.925819 11026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:42.925838 11026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:42.925854 11026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:42.925869 11026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:42.930976 11026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:43261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:43261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:42.931999 11026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:42.932931 11026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:42.941470 11031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:42.941494 11032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:42.941702 11026 server_base.cc:1034] running on GCE node
W20260925 22:46:42.941489 11034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:42.942269 11026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:42.943394 11026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:42.944595 11026 hybrid_clock.cc:648] HybridClock initialized: now 1790376402944571 us; error 33 us; skew 500 ppm
Sep 25 22:46:42 dist-test-slave-vdtf krb5kdc[11013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:42.947306 11026 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:46:42.948166 11026 webserver.cc:460] Webserver started at http://127.0.201.190:40585/ using document root <none> and password file <none>
I20260925 22:46:42.948681 11026 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:42.948741 11026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:42.948899 11026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:42.950863 11026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "9a6f3ff65f4a4cbabb67a91af463c299"
format_stamp: "Formatted at 2026-09-25 22:46:42 on dist-test-slave-vdtf"
server_key: "89f738d51f7080831a014a9f7cf13cb5"
server_key_iv: "908f9fd5517813c64bb43fcdfab99703"
server_key_version: "encryptionkey@0"
I20260925 22:46:42.951395 11026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "9a6f3ff65f4a4cbabb67a91af463c299"
format_stamp: "Formatted at 2026-09-25 22:46:42 on dist-test-slave-vdtf"
server_key: "89f738d51f7080831a014a9f7cf13cb5"
server_key_iv: "908f9fd5517813c64bb43fcdfab99703"
server_key_version: "encryptionkey@0"
I20260925 22:46:42.955062 11026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:46:42.957566 11041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:42.958818 11026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:42.958916 11026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "9a6f3ff65f4a4cbabb67a91af463c299"
format_stamp: "Formatted at 2026-09-25 22:46:42 on dist-test-slave-vdtf"
server_key: "89f738d51f7080831a014a9f7cf13cb5"
server_key_iv: "908f9fd5517813c64bb43fcdfab99703"
server_key_version: "encryptionkey@0"
I20260925 22:46:42.959048 11026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:42.978569 11026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:42.981659 11026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:42.981869 11026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:42.988585 11026 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:43261
I20260925 22:46:42.988865 11093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:43261 every 8 connection(s)
I20260925 22:46:42.989591 11026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:42.992420 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:42.997592 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 11026
I20260925 22:46:42.997720 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:46:42.997916 11094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299: Bootstrap starting.
I20260925 22:46:42.998023 806 external_mini_cluster.cc:1428] Setting key a3dd12ff355aaaa9302b60b556db169f
I20260925 22:46:43.000532 11094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:43.001376 11094 log.cc:824] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:43.003612 11094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299: No bootstrap required, opened a new log
I20260925 22:46:43.007462 11094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f3ff65f4a4cbabb67a91af463c299" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43261 } }
Sep 25 22:46:43 dist-test-slave-vdtf krb5kdc[11013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:43.007714 11094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:43.007786 11094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6f3ff65f4a4cbabb67a91af463c299, State: Initialized, Role: FOLLOWER
I20260925 22:46:43.008287 11094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f3ff65f4a4cbabb67a91af463c299" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43261 } }
I20260925 22:46:43.008455 11094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:43.008528 11094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:43.008636 11094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:43.009896 11094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f3ff65f4a4cbabb67a91af463c299" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43261 } }
I20260925 22:46:43.010262 11094 leader_election.cc:302] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a6f3ff65f4a4cbabb67a91af463c299; no voters:
I20260925 22:46:43.010613 11094 leader_election.cc:288] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:43.010795 11099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:43.011039 11099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [term 1 LEADER]: Becoming Leader. State: Replica: 9a6f3ff65f4a4cbabb67a91af463c299, State: Running, Role: LEADER
I20260925 22:46:43.011358 11099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f3ff65f4a4cbabb67a91af463c299" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43261 } }
I20260925 22:46:43.011992 11094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:43.013373 11101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a6f3ff65f4a4cbabb67a91af463c299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f3ff65f4a4cbabb67a91af463c299" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43261 } } }
I20260925 22:46:43.013490 11101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:43.013947 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:42.999445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.0.201.190:43261)
0925 22:46:42.999907 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:42.999926 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:42.999967 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:46:43.000832 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.000856 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.001282 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.001640 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:43.002517 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.004013 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.004870 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.005585 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.005742 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.008257 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:43.008263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:43.008287 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:43.008318 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:43.010368 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:43.010939 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:43.010946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:43.010952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:43.011190 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:43.011551 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:43.011554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:43.011556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:43.012065 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:43.012246 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.012677 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.012991 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":121,"threads_started":1}
I20260925 22:46:43.014518 11100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a6f3ff65f4a4cbabb67a91af463c299. Latest consensus state: current_term: 1 leader_uuid: "9a6f3ff65f4a4cbabb67a91af463c299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f3ff65f4a4cbabb67a91af463c299" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43261 } } }
I20260925 22:46:43.015331 11100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299 [sys.catalog]: This master's current role is: LEADER
W20260925 22:46:43.017545 11114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:46:43.017621 11114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:46:43.017745 11109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:43.018501 11109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:43.023604 11109 catalog_manager.cc:1347] Generated new cluster ID: ff408bcaed2a43ffb333e10d01efc644
I20260925 22:46:43.023667 11109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:43.033639 11109 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:43.034870 11109 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:43.052138 11109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299: Generated new TSK 0
I20260925 22:46:43.052913 11109 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:46:43.086552 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:43261
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:43.195395 11120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:43.195629 11120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:43.195715 11120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:43.199640 11120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:43.199712 11120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:43.199800 11120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:46:43.205226 11120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:43261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:43.206208 11120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:43.207185 11120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:43.216514 11125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:43.216526 11128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:43.216554 11126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:43.216799 11120 server_base.cc:1034] running on GCE node
I20260925 22:46:43.217134 11120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:43.217736 11120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:43.218911 11120 hybrid_clock.cc:648] HybridClock initialized: now 1790376403218888 us; error 46 us; skew 500 ppm
Sep 25 22:46:43 dist-test-slave-vdtf krb5kdc[11013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:43.221643 11120 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:46:43.222510 11120 webserver.cc:460] Webserver started at http://127.0.201.129:35299/ using document root <none> and password file <none>
I20260925 22:46:43.223070 11120 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:43.223161 11120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:43.223376 11120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:43.225383 11120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7"
format_stamp: "Formatted at 2026-09-25 22:46:43 on dist-test-slave-vdtf"
server_key: "f40d5d20e7591403d9bc13eb844bd710"
server_key_iv: "38213b475be923fe40ee91109fc91cd5"
server_key_version: "encryptionkey@0"
I20260925 22:46:43.225931 11120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7"
format_stamp: "Formatted at 2026-09-25 22:46:43 on dist-test-slave-vdtf"
server_key: "f40d5d20e7591403d9bc13eb844bd710"
server_key_iv: "38213b475be923fe40ee91109fc91cd5"
server_key_version: "encryptionkey@0"
I20260925 22:46:43.229553 11120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:43.231936 11135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:43.233191 11120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:43.233304 11120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7"
format_stamp: "Formatted at 2026-09-25 22:46:43 on dist-test-slave-vdtf"
server_key: "f40d5d20e7591403d9bc13eb844bd710"
server_key_iv: "38213b475be923fe40ee91109fc91cd5"
server_key_version: "encryptionkey@0"
I20260925 22:46:43.233412 11120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:43.244652 11120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:43.247350 11120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:43.247488 11120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:43.248034 11120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:43.248750 11120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:43.248796 11120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:43.248832 11120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:43.248848 11120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:43.258009 11120 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:45941
I20260925 22:46:43.258034 11248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:45941 every 8 connection(s)
I20260925 22:46:43.258911 11120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:43.262652 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 11120
I20260925 22:46:43.262811 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:43.263154 806 external_mini_cluster.cc:1428] Setting key de27770acd733e29f39639c1ae61fd3a
Sep 25 22:46:43 dist-test-slave-vdtf krb5kdc[11013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:46:43.273779 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.260693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:41335 (local address 127.0.201.190:43261)
0925 22:46:43.260820 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:43.260824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:43.261846 (+ 1022us) server_negotiation.cc:407] Connection header received
0925 22:46:43.262737 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.262741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.262794 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.262903 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:43.264265 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.264917 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.266655 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.267054 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.267180 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.270047 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:43.270050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:43.270052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:43.270075 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:43.271488 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:43.272119 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:43.272122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:43.272123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:43.272194 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:43.272590 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:43.272592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:43.272593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:43.272765 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:43.272842 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.273513 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.273659 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:46:43.274539 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.261049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:43261 (local address 127.0.201.129:41335)
0925 22:46:43.261703 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:43.261744 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:43.262520 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:43.263167 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:43.263174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:43.263556 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:43.264128 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.264141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.265068 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.265074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.266517 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.266527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.267201 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.267203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.267357 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.267993 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:43.268009 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:43.269873 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:43.271619 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:43.271624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:43.271638 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:43.272024 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:43.272290 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:43.272296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:43.272301 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:43.272488 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:43.272877 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:43.272883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:43.273175 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 22:46:43.273433 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 22:46:43.273677 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":108,"threads_started":1}
I20260925 22:46:43.275877 11249 heartbeater.cc:347] Connected to a master server at 127.0.201.190:43261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:46:43.276144 11249 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:43.276711 11249 heartbeater.cc:511] Master 127.0.201.190:43261 requested a full tablet report, sending...
I20260925 22:46:43.278254 11058 ts_manager.cc:194] Registered new tserver with Master: 9b90cd7c7b8e4462af144c1c0e78f2c7 (127.0.201.129:45941)
I20260925 22:46:43.279342 11058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:41335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:46:43.290890 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:43261
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:43.399276 11254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:43.399608 11254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:43.399742 11254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:43.403718 11254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:43.403811 11254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:43.403903 11254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:46:43.409029 11254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:43261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:43.410053 11254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:43.411053 11254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:43.420728 11262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:43.420799 11259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:43.420792 11260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:43.420956 11254 server_base.cc:1034] running on GCE node
I20260925 22:46:43.421452 11254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:43.422070 11254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:43.423264 11254 hybrid_clock.cc:648] HybridClock initialized: now 1790376403423241 us; error 26 us; skew 500 ppm
Sep 25 22:46:43 dist-test-slave-vdtf krb5kdc[11013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:43.426044 11254 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:46:43.426908 11254 webserver.cc:460] Webserver started at http://127.0.201.130:40489/ using document root <none> and password file <none>
I20260925 22:46:43.427457 11254 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:43.427548 11254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:43.427855 11254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:43.429840 11254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "75cc8537e5184cf998085197e1444aaa"
format_stamp: "Formatted at 2026-09-25 22:46:43 on dist-test-slave-vdtf"
server_key: "e9ffca6755b8e414abb6d7a748d701e1"
server_key_iv: "d3758b3f03b5844f62a418ea05d94257"
server_key_version: "encryptionkey@0"
I20260925 22:46:43.430402 11254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "75cc8537e5184cf998085197e1444aaa"
format_stamp: "Formatted at 2026-09-25 22:46:43 on dist-test-slave-vdtf"
server_key: "e9ffca6755b8e414abb6d7a748d701e1"
server_key_iv: "d3758b3f03b5844f62a418ea05d94257"
server_key_version: "encryptionkey@0"
I20260925 22:46:43.433986 11254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:43.436415 11269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:43.437676 11254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:46:43.437813 11254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "75cc8537e5184cf998085197e1444aaa"
format_stamp: "Formatted at 2026-09-25 22:46:43 on dist-test-slave-vdtf"
server_key: "e9ffca6755b8e414abb6d7a748d701e1"
server_key_iv: "d3758b3f03b5844f62a418ea05d94257"
server_key_version: "encryptionkey@0"
I20260925 22:46:43.437932 11254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:43.456382 11254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:43.459057 11254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:43.459249 11254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:43.459841 11254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:43.460620 11254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:43.460690 11254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:43.460758 11254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:43.460810 11254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:43.469162 11254 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:39667
I20260925 22:46:43.469180 11382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:39667 every 8 connection(s)
I20260925 22:46:43.469991 11254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:43.476783 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 11254
I20260925 22:46:43.476877 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:43.477170 806 external_mini_cluster.cc:1428] Setting key c3d5e04d7f92ce3e819cfd8d62fd2bcb
Sep 25 22:46:43 dist-test-slave-vdtf krb5kdc[11013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:43.483806 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.471732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:42223 (local address 127.0.201.190:43261)
0925 22:46:43.471893 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:43.471898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:43.472977 (+ 1079us) server_negotiation.cc:407] Connection header received
0925 22:46:43.473785 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.473788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.473834 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.473912 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:43.475005 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.475533 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.476775 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.477169 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.477308 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.480136 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:43.480143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:43.480148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:43.480182 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:43.481780 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:43.482284 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:43.482287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:43.482289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:43.482371 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:43.482711 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:43.482714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:43.482715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:43.482894 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:43.483002 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.483478 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.483606 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:46:43.484634 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.472193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:43261 (local address 127.0.201.130:42223)
0925 22:46:43.472827 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:43.472866 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:43.473591 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:43.474068 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:43.474074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:43.474405 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:43.474871 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.474883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.475633 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.475635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.476640 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.476653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.477370 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.477374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.477563 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.478176 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:43.478191 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:43.479936 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:43.481893 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:43.481899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:43.481914 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:43.482176 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:43.482465 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:43.482470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:43.482472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:43.482611 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:43.482998 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:43.483004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:43.483247 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 22:46:43.483426 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 22:46:43.483759 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":120,"threads_started":1}
I20260925 22:46:43.485880 11383 heartbeater.cc:347] Connected to a master server at 127.0.201.190:43261
I20260925 22:46:43.486119 11383 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:43.486608 11383 heartbeater.cc:511] Master 127.0.201.190:43261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:46:43.487632 11058 ts_manager.cc:194] Registered new tserver with Master: 75cc8537e5184cf998085197e1444aaa (127.0.201.130:39667)
I20260925 22:46:43.488235 11058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:42223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:46:43.505009 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:43261
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:43.616174 11388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:43.616405 11388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:43.616447 11388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:43.620431 11388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:43.620489 11388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:43.620553 11388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:46:43.625639 11388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:43261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:46:43.626646 11388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:43.627632 11388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:43.637199 11396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:43.637382 11388 server_base.cc:1034] running on GCE node
W20260925 22:46:43.637207 11394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:43.637200 11393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:43.637912 11388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:43.638646 11388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:43.639743 11388 hybrid_clock.cc:648] HybridClock initialized: now 1790376403639712 us; error 48 us; skew 500 ppm
Sep 25 22:46:43 dist-test-slave-vdtf krb5kdc[11013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:43.642326 11388 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:46:43.643092 11388 webserver.cc:460] Webserver started at http://127.0.201.131:34935/ using document root <none> and password file <none>
I20260925 22:46:43.643581 11388 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:43.643637 11388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:43.643855 11388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:46:43.645720 11388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "5e8ae5c856b3469e87fb88b79fa33a1b"
format_stamp: "Formatted at 2026-09-25 22:46:43 on dist-test-slave-vdtf"
server_key: "e4518ac742f9007688bc125cc17edc81"
server_key_iv: "79c31ff1d3609482832aceabb3e08fdb"
server_key_version: "encryptionkey@0"
I20260925 22:46:43.646201 11388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "5e8ae5c856b3469e87fb88b79fa33a1b"
format_stamp: "Formatted at 2026-09-25 22:46:43 on dist-test-slave-vdtf"
server_key: "e4518ac742f9007688bc125cc17edc81"
server_key_iv: "79c31ff1d3609482832aceabb3e08fdb"
server_key_version: "encryptionkey@0"
I20260925 22:46:43.649669 11388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:46:43.651901 11403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:43.653185 11388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:46:43.653282 11388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "5e8ae5c856b3469e87fb88b79fa33a1b"
format_stamp: "Formatted at 2026-09-25 22:46:43 on dist-test-slave-vdtf"
server_key: "e4518ac742f9007688bc125cc17edc81"
server_key_iv: "79c31ff1d3609482832aceabb3e08fdb"
server_key_version: "encryptionkey@0"
I20260925 22:46:43.653360 11388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:43.668761 11388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:43.671768 11388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:43.671970 11388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:43.672555 11388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:43.673360 11388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:43.673405 11388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:43.673472 11388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:43.673518 11388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:43.681895 11388 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:39179
I20260925 22:46:43.681960 11516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:39179 every 8 connection(s)
I20260925 22:46:43.682801 11388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:43.691285 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 11388
I20260925 22:46:43.691450 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
Sep 25 22:46:43 dist-test-slave-vdtf krb5kdc[11013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:46:43.692230 806 external_mini_cluster.cc:1428] Setting key ce7ba0ed68d32a5ca2963876eb54f6ab
I20260925 22:46:43.701018 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.684619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:43985 (local address 127.0.201.190:43261)
0925 22:46:43.684779 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:43.684784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:43.685756 (+ 972us) server_negotiation.cc:407] Connection header received
0925 22:46:43.686588 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.686590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.686646 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.686720 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:43.687883 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.688396 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.689469 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.689693 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.689772 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.692465 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:43.692470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:43.692474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:43.692504 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:43.699226 (+ 6722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:43.699739 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:43.699742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:43.699744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:43.699826 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:43.700136 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:43.700138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:43.700140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:43.700307 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:43.700381 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.700814 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.700904 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:46:43.701836 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.684991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:43261 (local address 127.0.201.131:43985)
0925 22:46:43.685617 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:43.685657 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:43.686401 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:43.686875 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:43.686882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:43.687241 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:43.687762 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.687778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.688495 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.688498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.689366 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.689374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.689819 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.689822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.689951 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.690479 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:43.690495 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:43.692281 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:43.699331 (+ 7050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:43.699336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:43.699347 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:43.699606 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:43.699905 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:43.699907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:43.699909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:43.700035 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:43.700390 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:43.700395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:43.700619 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 22:46:43.700769 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 22:46:43.701001 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":111,"threads_started":1}
I20260925 22:46:43.702932 11517 heartbeater.cc:347] Connected to a master server at 127.0.201.190:43261
I20260925 22:46:43.703150 11517 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:43.703560 11517 heartbeater.cc:511] Master 127.0.201.190:43261 requested a full tablet report, sending...
I20260925 22:46:43.704528 11058 ts_manager.cc:194] Registered new tserver with Master: 5e8ae5c856b3469e87fb88b79fa33a1b (127.0.201.131:39179)
I20260925 22:46:43.705157 11058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:43985
I20260925 22:46:43.706244 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:46:43.714637 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.708034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.0.201.190:43261)
0925 22:46:43.708151 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:43.708155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:43.708343 (+ 188us) server_negotiation.cc:407] Connection header received
0925 22:46:43.708427 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.708430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.708462 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.708520 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:43.709107 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.709569 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.710363 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.710556 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.710636 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.711858 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:43.711861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:43.711863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:43.711882 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:43.713261 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:43.713693 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:43.713696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:43.713697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:43.713760 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:43.713988 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:43.713990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:43.713991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:43.714172 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:43.714269 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.714447 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.714528 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:46:43.717139 11058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:46:43.720093 11058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:43.734123 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.728250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.0.201.130:39667)
0925 22:46:43.728636 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:43.728651 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:43.728677 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:46:43.728929 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.728935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.729513 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.729683 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:43.729971 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.731138 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.732911 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.733456 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.733587 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.733727 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.733870 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.733964 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":115,"threads_started":1}
I20260925 22:46:43.734150 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.728135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:39667 (local address 127.0.0.1:60352)
0925 22:46:43.728560 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:43.728603 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:43.728721 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:43.729694 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:43.729701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:43.729725 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:43.729853 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.729859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.731271 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.731274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.732786 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.732796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.733584 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.733587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.733685 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.733718 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 22:46:43.733824 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 22:46:43.733964 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":76,"thread_start_us":76,"threads_started":1}
I20260925 22:46:43.734760 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.729177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.0.201.131:39179)
0925 22:46:43.729721 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:43.729733 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:43.729754 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:46:43.729817 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.729821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.729996 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.730153 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:43.730567 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.731294 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.732989 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.733501 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.733631 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.733765 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.734518 (+ 753us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.734634 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":126,"threads_started":1}
I20260925 22:46:43.735036 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.728212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:45941 (local address 127.0.0.1:43312)
0925 22:46:43.728769 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:43.728782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:43.728866 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:43.730373 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:43.730378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:43.730396 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:43.730467 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.730470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.731763 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.731768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.732781 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.732789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.734678 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.734681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.734769 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.734788 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:46:43.734850 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:46:43.734908 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":6,"thread_start_us":94,"threads_started":1}
I20260925 22:46:43.735347 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.729187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.0.201.129:45941)
0925 22:46:43.729956 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:43.729972 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:43.729992 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:46:43.730064 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.730069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.730234 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.730375 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:43.730548 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.731585 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.733787 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.734558 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.734689 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.734942 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.734995 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.735216 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":101,"threads_started":1}
I20260925 22:46:43.737254 11317 tablet_service.cc:1467] Processing CreateTablet for tablet d23ba9046c364cbe85def01542217a86 (DEFAULT_TABLE table=test-table [id=5133bc03fc804e278dfa5240e1626c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:43.737556 11183 tablet_service.cc:1467] Processing CreateTablet for tablet d23ba9046c364cbe85def01542217a86 (DEFAULT_TABLE table=test-table [id=5133bc03fc804e278dfa5240e1626c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:43.738220 11317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23ba9046c364cbe85def01542217a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:43.738288 11183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23ba9046c364cbe85def01542217a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:43.739724 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.728818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:39179 (local address 127.0.0.1:59170)
0925 22:46:43.729231 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:43.729244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:43.729331 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:43.730138 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:43.730141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:43.730157 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:43.730254 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.730264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.731454 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.731457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.732905 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.732912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.733631 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.733634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.733723 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.733744 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:46:43.733827 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:46:43.733957 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":170,"thread_start_us":82,"threads_started":1}
I20260925 22:46:43.742393 11451 tablet_service.cc:1467] Processing CreateTablet for tablet d23ba9046c364cbe85def01542217a86 (DEFAULT_TABLE table=test-table [id=5133bc03fc804e278dfa5240e1626c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:43.743404 11451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23ba9046c364cbe85def01542217a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:43.745025 11534 tablet_bootstrap.cc:492] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: Bootstrap starting.
I20260925 22:46:43.746243 11535 tablet_bootstrap.cc:492] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7: Bootstrap starting.
I20260925 22:46:43.747414 11534 tablet_bootstrap.cc:654] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:43.748181 11534 log.cc:824] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:43.748306 11535 tablet_bootstrap.cc:654] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:43.749488 11535 log.cc:824] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:43.750188 11534 tablet_bootstrap.cc:492] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: No bootstrap required, opened a new log
I20260925 22:46:43.750375 11534 ts_tablet_manager.cc:1397] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:46:43.750399 11538 tablet_bootstrap.cc:492] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: Bootstrap starting.
I20260925 22:46:43.751166 11535 tablet_bootstrap.cc:492] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7: No bootstrap required, opened a new log
I20260925 22:46:43.751362 11535 ts_tablet_manager.cc:1397] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:46:43.752521 11538 tablet_bootstrap.cc:654] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:43.753662 11538 log.cc:824] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:43.754401 11535 raft_consensus.cc:348] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cc8537e5184cf998085197e1444aaa" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39667 } } peers { permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 } } peers { permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 } }
I20260925 22:46:43.754444 11534 raft_consensus.cc:348] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cc8537e5184cf998085197e1444aaa" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39667 } } peers { permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 } } peers { permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 } }
I20260925 22:46:43.754643 11535 raft_consensus.cc:374] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:43.754695 11535 raft_consensus.cc:729] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b90cd7c7b8e4462af144c1c0e78f2c7, State: Initialized, Role: FOLLOWER
I20260925 22:46:43.754709 11534 raft_consensus.cc:374] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:43.754753 11534 raft_consensus.cc:729] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cc8537e5184cf998085197e1444aaa, State: Initialized, Role: FOLLOWER
I20260925 22:46:43.755132 11535 consensus_queue.cc:260] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cc8537e5184cf998085197e1444aaa" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39667 } } peers { permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 } } peers { permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 } }
I20260925 22:46:43.755183 11534 consensus_queue.cc:260] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cc8537e5184cf998085197e1444aaa" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39667 } } peers { permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 } } peers { permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 } }
I20260925 22:46:43.755527 11538 tablet_bootstrap.cc:492] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: No bootstrap required, opened a new log
I20260925 22:46:43.755714 11538 ts_tablet_manager.cc:1397] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:46:43.755788 11249 heartbeater.cc:503] Master 127.0.201.190:43261 was elected leader, sending a full tablet report...
I20260925 22:46:43.756063 11535 ts_tablet_manager.cc:1428] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:46:43.756158 11534 ts_tablet_manager.cc:1428] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:46:43.756482 11383 heartbeater.cc:503] Master 127.0.201.190:43261 was elected leader, sending a full tablet report...
W20260925 22:46:43.759997 11250 tablet.cc:2369] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:43.760113 11538 raft_consensus.cc:348] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cc8537e5184cf998085197e1444aaa" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39667 } } peers { permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 } } peers { permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 } }
I20260925 22:46:43.760358 11538 raft_consensus.cc:374] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:43.760417 11538 raft_consensus.cc:729] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8ae5c856b3469e87fb88b79fa33a1b, State: Initialized, Role: FOLLOWER
I20260925 22:46:43.760785 11538 consensus_queue.cc:260] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cc8537e5184cf998085197e1444aaa" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39667 } } peers { permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 } } peers { permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 } }
I20260925 22:46:43.761395 11517 heartbeater.cc:503] Master 127.0.201.190:43261 was elected leader, sending a full tablet report...
I20260925 22:46:43.761467 11541 raft_consensus.cc:482] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:43.761610 11541 raft_consensus.cc:504] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cc8537e5184cf998085197e1444aaa" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39667 } } peers { permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 } } peers { permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 } }
I20260925 22:46:43.761727 11538 ts_tablet_manager.cc:1428] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:46:43.762743 11541 leader_election.cc:288] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b90cd7c7b8e4462af144c1c0e78f2c7 (127.0.201.129:45941), 5e8ae5c856b3469e87fb88b79fa33a1b (127.0.201.131:39179)
I20260925 22:46:43.766572 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.763051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:37417 (local address 127.0.201.129:45941)
0925 22:46:43.763199 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:43.763203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:43.763217 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:46:43.763284 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.763290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.763369 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.763457 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:43.763841 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.764554 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.765662 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.766049 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.766172 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.766222 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.766395 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.766455 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:46:43.766572 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.762977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:45941 (local address 127.0.201.130:37417)
0925 22:46:43.763100 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:43.763118 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:43.763196 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:43.763571 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:43.763574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:43.763592 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:43.763742 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.763751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.764678 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.764682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.765530 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.765539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.766161 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.766164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.766264 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.766286 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:46:43.766382 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:46:43.766451 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"spinlock_wait_cycles":768}
I20260925 22:46:43.766922 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.763600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:55681 (local address 127.0.201.131:39179)
0925 22:46:43.763734 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:43.763738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:43.763751 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:46:43.763792 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.763796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.763843 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.763908 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:43.764305 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.764990 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.765975 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.766363 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.766481 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.766525 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.766752 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.766819 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:46:43.767156 11203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23ba9046c364cbe85def01542217a86" candidate_uuid: "75cc8537e5184cf998085197e1444aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" is_pre_election: true
I20260925 22:46:43.767266 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.763240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:39179 (local address 127.0.201.130:55681)
0925 22:46:43.763517 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:43.763530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:43.763637 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:43.764043 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:43.764047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:43.764065 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:43.764179 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.764185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.765096 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.765098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.765850 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:43.765857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.766474 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.766478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:43.766578 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.766599 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:46:43.767052 (+ 453us) client_negotiation.cc:240] Negotiation successful
0925 22:46:43.767121 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"spinlock_wait_cycles":4736,"thread_start_us":92,"threads_started":1}
I20260925 22:46:43.767401 11203 raft_consensus.cc:2454] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cc8537e5184cf998085197e1444aaa in term 0.
I20260925 22:46:43.767871 11273 leader_election.cc:302] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75cc8537e5184cf998085197e1444aaa, 9b90cd7c7b8e4462af144c1c0e78f2c7; no voters:
I20260925 22:46:43.767867 11471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23ba9046c364cbe85def01542217a86" candidate_uuid: "75cc8537e5184cf998085197e1444aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" is_pre_election: true
I20260925 22:46:43.768110 11471 raft_consensus.cc:2454] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cc8537e5184cf998085197e1444aaa in term 0.
I20260925 22:46:43.768110 11541 raft_consensus.cc:2781] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:43.768199 11541 raft_consensus.cc:482] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:43.768268 11541 raft_consensus.cc:3037] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:43.769232 11541 raft_consensus.cc:504] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cc8537e5184cf998085197e1444aaa" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39667 } } peers { permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 } } peers { permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 } }
I20260925 22:46:43.769541 11541 leader_election.cc:288] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [CANDIDATE]: Term 1 election: Requested vote from peers 9b90cd7c7b8e4462af144c1c0e78f2c7 (127.0.201.129:45941), 5e8ae5c856b3469e87fb88b79fa33a1b (127.0.201.131:39179)
I20260925 22:46:43.769914 11203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23ba9046c364cbe85def01542217a86" candidate_uuid: "75cc8537e5184cf998085197e1444aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7"
I20260925 22:46:43.770005 11471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23ba9046c364cbe85def01542217a86" candidate_uuid: "75cc8537e5184cf998085197e1444aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b"
I20260925 22:46:43.770043 11203 raft_consensus.cc:3037] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:43.770129 11471 raft_consensus.cc:3037] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:43.771024 11471 raft_consensus.cc:2454] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75cc8537e5184cf998085197e1444aaa in term 1.
I20260925 22:46:43.771032 11203 raft_consensus.cc:2454] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75cc8537e5184cf998085197e1444aaa in term 1.
I20260925 22:46:43.771267 11273 leader_election.cc:302] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8ae5c856b3469e87fb88b79fa33a1b, 75cc8537e5184cf998085197e1444aaa; no voters:
I20260925 22:46:43.771528 11541 raft_consensus.cc:2781] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:43.771762 11541 raft_consensus.cc:686] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 1 LEADER]: Becoming Leader. State: Replica: 75cc8537e5184cf998085197e1444aaa, State: Running, Role: LEADER
I20260925 22:46:43.772069 11541 consensus_queue.cc:237] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cc8537e5184cf998085197e1444aaa" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39667 } } peers { permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 } } peers { permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 } }
I20260925 22:46:43.774967 11058 catalog_manager.cc:5495] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 75cc8537e5184cf998085197e1444aaa (127.0.201.130). New cstate: current_term: 1 leader_uuid: "75cc8537e5184cf998085197e1444aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cc8537e5184cf998085197e1444aaa" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 } health_report { overall_health: UNKNOWN } } }
I20260925 22:46:43.795727 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.792234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.0.201.130:39667)
0925 22:46:43.792390 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:43.792395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:43.792408 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:46:43.792479 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:43.792481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:43.792537 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:43.792648 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:43.792879 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.793378 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.794250 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:43.794466 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:43.794581 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:43.794889 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:43.795291 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:43.795422 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:43.795483 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 22:46:43.795542 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:46:43.801201 11471 raft_consensus.cc:1261] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b [term 1 FOLLOWER]: Refusing update from remote peer 75cc8537e5184cf998085197e1444aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:43.801214 11203 raft_consensus.cc:1261] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7 [term 1 FOLLOWER]: Refusing update from remote peer 75cc8537e5184cf998085197e1444aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:43.801893 11541 consensus_queue.cc:1035] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b90cd7c7b8e4462af144c1c0e78f2c7" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 45941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:43.802110 11544 consensus_queue.cc:1035] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8ae5c856b3469e87fb88b79fa33a1b" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 39179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:43.810514 11549 mvcc.cc:205] Tried to move back new op lower bound from 7333381749962973184 to 7333381749853270016. Current Snapshot: MvccSnapshot[applied={T|T < 7333381749962973184}]
I20260925 22:46:43.816960 11057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:43.817093 11057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:43.819289 11057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a6f3ff65f4a4cbabb67a91af463c299: Sending DeleteTablet for 3 replicas of tablet d23ba9046c364cbe85def01542217a86
I20260925 22:46:43.819965 11317 tablet_service.cc:1514] Processing DeleteTablet for tablet d23ba9046c364cbe85def01542217a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:43 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:60352
I20260925 22:46:43.820241 11183 tablet_service.cc:1514] Processing DeleteTablet for tablet d23ba9046c364cbe85def01542217a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:43 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:43312
I20260925 22:46:43.820392 11559 tablet_replica.cc:330] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: stopping tablet replica
I20260925 22:46:43.820665 11559 raft_consensus.cc:2229] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:46:43.820686 11560 tablet_replica.cc:330] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7: stopping tablet replica
I20260925 22:46:43.820694 11451 tablet_service.cc:1514] Processing DeleteTablet for tablet d23ba9046c364cbe85def01542217a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:43 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:59170
I20260925 22:46:43.820927 11560 raft_consensus.cc:2229] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:43.821007 11559 raft_consensus.cc:2258] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:43.821177 11560 raft_consensus.cc:2258] T d23ba9046c364cbe85def01542217a86 P 9b90cd7c7b8e4462af144c1c0e78f2c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:43.821631 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 11120
I20260925 22:46:43.824944 11559 ts_tablet_manager.cc:1905] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:43.827482 11561 tablet_replica.cc:330] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: stopping tablet replica
I20260925 22:46:43.827880 11561 raft_consensus.cc:2229] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:43.828240 11561 raft_consensus.cc:2258] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:43.828554 11559 ts_tablet_manager.cc:1918] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:43.829111 11559 log.cc:1196] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/d23ba9046c364cbe85def01542217a86
I20260925 22:46:43.829486 11559 ts_tablet_manager.cc:1939] T d23ba9046c364cbe85def01542217a86 P 75cc8537e5184cf998085197e1444aaa: Deleting consensus metadata
I20260925 22:46:43.829903 11561 ts_tablet_manager.cc:1905] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:46:43.830164 11045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:45941 (error 108)
I20260925 22:46:43.830770 11043 catalog_manager.cc:4841] TS 75cc8537e5184cf998085197e1444aaa (127.0.201.130:39667): tablet d23ba9046c364cbe85def01542217a86 (table test-table [id=5133bc03fc804e278dfa5240e1626c9b]) successfully deleted
I20260925 22:46:43.831110 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 11254
I20260925 22:46:43.833251 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:43.832959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:45941 (local address 127.0.0.1:43314)
0925 22:46:43.833090 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:43.833157 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:46:43.833732 11045 catalog_manager.cc:4584] TS 9b90cd7c7b8e4462af144c1c0e78f2c7 (127.0.201.129:45941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d23ba9046c364cbe85def01542217a86: Network error: Client connection negotiation failed: client connection to 127.0.201.129:45941: connect: Connection refused (error 111)
I20260925 22:46:43.833771 11561 ts_tablet_manager.cc:1918] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:43.833858 11561 log.cc:1196] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/d23ba9046c364cbe85def01542217a86
I20260925 22:46:43.834180 11561 ts_tablet_manager.cc:1939] T d23ba9046c364cbe85def01542217a86 P 5e8ae5c856b3469e87fb88b79fa33a1b: Deleting consensus metadata
I20260925 22:46:43.835431 11045 catalog_manager.cc:4841] TS 5e8ae5c856b3469e87fb88b79fa33a1b (127.0.201.131:39179): tablet d23ba9046c364cbe85def01542217a86 (table test-table [id=5133bc03fc804e278dfa5240e1626c9b]) successfully deleted
I20260925 22:46:43.838546 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 11388
I20260925 22:46:43.846498 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 11026
2026-09-25T22:46:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2217 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:46:43 dist-test-slave-vdtf krb5kdc[11565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:46:43 dist-test-slave-vdtf krb5kdc[11565](info): set up 2 sockets
Sep 25 22:46:43 dist-test-slave-vdtf krb5kdc[11565](info): commencing operation
krb5kdc: starting...
W20260925 22:46:44.892721 806 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:46:44 dist-test-slave-vdtf krb5kdc[11565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:46:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.201.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.201.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/rangeradmin_127.0.201.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/rangeradmin_127.0.201.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.201.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.201.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/rangerlookup_127.0.201.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/rangerlookup_127.0.201.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.201.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.201.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/HTTP_127.0.201.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/HTTP_127.0.201.148@KRBTEST.COM.keytab.
I20260925 22:46:45.054801 806 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrAtchr/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/postgres -l logfile start
2026-09-25 22:46:47.528 UTC [11595] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:46:47.528 UTC [11595] LOG: listening on IPv4 address "127.0.201.148", port 54213
2026-09-25 22:46:47.531 UTC [11595] LOG: listening on Unix socket "/tmp/.s.PGSQL.54213"
2026-09-25 22:46:47.555 UTC [11598] LOG: database system was shut down at 2026-09-25 22:46:46 UTC
2026-09-25 22:46:47.558 UTC [11595] LOG: database system is ready to accept connections
I20260925 22:46:48.520635 806 mini_postgres.cc:96] Postgres bound to 54213
2026-09-25 22:46:48.527 UTC [11606] FATAL: database "slave" does not exist
127.0.201.148:54213 - accepting connections
I20260925 22:46:48.527872 806 mini_ranger.cc:161] Starting Ranger...
I20260925 22:46:48.542603 806 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:46:49.023416 806 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:46:49.023523 806 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-admin
2026-09-25 22:46:49,306 [I] DB FLAVOR :POSTGRES
2026-09-25 22:46:49,306 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:46:49,306 [I] Checking connection..
2026-09-25 22:46:49,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:46:49,545 [I] Checking connection passed.
2026-09-25 22:46:49,545 [I] --------- Verifying version history table ---------
2026-09-25 22:46:49,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:46:49,789 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:46:49,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:46:50,041 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:46:50,041 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:46:50,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:46:50.270 UTC [11670] WARNING: there is no transaction in progress
2026-09-25 22:46:50,276 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:46:50,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:46:50,501 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:46:50,501 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:46:50,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:46:50,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:46:50,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vdtf.c.gcp-upstream.internal','N') ;"
2026-09-25 22:46:51,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:46:51,445 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:46:51,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:46:51,705 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:46:51,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:46:51,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:46:52,184 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:46:52,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:46:53.145 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.161 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.179 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.195 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.211 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.229 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.292 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.299 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.313 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.324 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.334 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.348 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.357 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.363 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.370 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:53.377 UTC [11778] WARNING: there is no transaction in progress
2026-09-25 22:46:54,545 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:46:54,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:46:54,764 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:46:54,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:46:54,993 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:46:54,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:46:55,252 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:46:55,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:46:55,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:46:55,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vdtf.c.gcp-upstream.internal';"
2026-09-25 22:46:55,973 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:46:55,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:46:56,207 [I] DB_PATCHES have already been applied
I20260925 22:46:56.213989 806 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrAtchr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:46:56.214056 806 mini_ranger.cc:193] Using host: 127.0.201.148
I20260925 22:46:56.217626 806 mini_ranger.cc:227] Ranger admin URL: http://127.0.201.148:45887
Sep 25, 2026 10:46:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:46:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:46:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:46:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:46:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:46:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:46:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:46:57.346273 806 mini_ranger.cc:160] Time spent starting Ranger: real 8.818s user 0.002s sys 0.008s
Sep 25, 2026 10:47:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:47:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 10:47:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:47:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:47:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:47:17.645356 806 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.201.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.201.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/rangerkms_127.0.201.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/rangerkms_127.0.201.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.201.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.201.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.201.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/HTTP_127.0.201.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.201.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/HTTP_127.0.201.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:47:17 dist-test-slave-vdtf krb5kdc[11565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376437, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:47:17.750339 806 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:47:17.760895 806 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:47:18.710176 806 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:47:18.710273 806 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-kms
I20260925 22:47:18.710371 806 mini_ranger_kms.cc:227] Using postgres at 127.0.201.148:54213
2026-09-25 22:47:18,895 [I] DB FLAVOR :POSTGRES
2026-09-25 22:47:18,896 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:47:18,896 [I] Checking connection
2026-09-25 22:47:18,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:47:19,147 [I] connection success
2026-09-25 22:47:19,148 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:47:19,148 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:47:19,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:47:19,375 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:47:19,375 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:47:19,376 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:47:19,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:54213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:47:19,640 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:47:20.097286 806 mini_ranger_kms.cc:310] Created kms service
I20260925 22:47:20.291855 806 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:47:20.332240 806 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:47:20 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376437, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:47:20.705143 806 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:47:20.705302 806 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:47:20.705328 806 mini_ranger_kms.cc:240] Using host: 127.0.201.148
I20260925 22:47:20.708838 806 mini_ranger_kms.cc:276] Ranger KMS PID: 11958
I20260925 22:47:20.708978 806 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.201.148:54275
Sep 25, 2026 10:47:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:47:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:47:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:47:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:47:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:47:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 22:47:22.935995 806 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.186s user 0.018s sys 0.151s
I20260925 22:47:22.936127 806 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:47:22.936234 806 mini_ranger_kms.cc:401] 127.0.201.148:54275/kms/v1/keys
Sep 25 22:47:23 dist-test-slave-vdtf krb5kdc[11565](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376443, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.201.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 22:47:23 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376443, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.201.148@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
Sep 25 22:47:23 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376443, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.201.148@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
Sep 25 22:47:26 dist-test-slave-vdtf krb5kdc[11565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:47:26.065158 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:41517
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:41517
--ranger_config_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:54275/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:26.177177 12009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:26.177429 12009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:26.177469 12009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:26.181509 12009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:26.181560 12009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:26.181581 12009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:26.181596 12009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:26.181612 12009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:26.186930 12009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.201.148:54275/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:41517
--ranger_config_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:47:26.188000 12009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:26.189043 12009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:26.198741 12014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:26.198781 12015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:26.198936 12009 server_base.cc:1034] running on GCE node
W20260925 22:47:26.199105 12017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:26.199747 12009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:26.200851 12009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:26.202034 12009 hybrid_clock.cc:648] HybridClock initialized: now 1790376446202008 us; error 49 us; skew 500 ppm
Sep 25 22:47:26 dist-test-slave-vdtf krb5kdc[11565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:26.210340 12009 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:47:26.211447 12009 webserver.cc:460] Webserver started at http://127.0.201.190:43653/ using document root <none> and password file <none>
I20260925 22:47:26.212072 12009 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:26.212129 12009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:26.212360 12009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:47:26 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:47:26.259332 12009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "6065528665154b95b4290989ac791e69"
format_stamp: "Formatted at 2026-09-25 22:47:26 on dist-test-slave-vdtf"
server_key: "291832f8dc3a0de6f31f3cc02bac5ebe"
server_key_iv: "82531dd5b5b2a87aa0d263929b000904"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:26.260044 12009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "6065528665154b95b4290989ac791e69"
format_stamp: "Formatted at 2026-09-25 22:47:26 on dist-test-slave-vdtf"
server_key: "291832f8dc3a0de6f31f3cc02bac5ebe"
server_key_iv: "82531dd5b5b2a87aa0d263929b000904"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:26.264267 12009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 22:47:26.281155 12024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:26.282609 12009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:47:26.282742 12009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "6065528665154b95b4290989ac791e69"
format_stamp: "Formatted at 2026-09-25 22:47:26 on dist-test-slave-vdtf"
server_key: "291832f8dc3a0de6f31f3cc02bac5ebe"
server_key_iv: "82531dd5b5b2a87aa0d263929b000904"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:26.282864 12009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:26.298751 12009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:26.301627 12009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:26.301818 12009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:26.310381 12009 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:41517
I20260925 22:47:26.310439 12076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:41517 every 8 connection(s)
I20260925 22:47:26.311460 12009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:47:26.312812 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 12009
I20260925 22:47:26.312945 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:47:26.313257 806 external_mini_cluster.cc:1428] Setting key 033218d2f61027ccd93516ea01867494
I20260925 22:47:26.314986 12077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:26.321619 12077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69: Bootstrap starting.
I20260925 22:47:26.324221 12077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69: Neither blocks nor log segments found. Creating new log.
Sep 25 22:47:26 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:47:26.325156 12077 log.cc:824] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:26.327389 12077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69: No bootstrap required, opened a new log
I20260925 22:47:26.330767 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:26.314731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.0.201.190:41517)
0925 22:47:26.315213 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:26.315231 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:26.316022 (+ 791us) server_negotiation.cc:407] Connection header received
0925 22:47:26.316957 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:26.316979 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:26.317419 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:26.317791 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:26.318687 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:26.320198 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:26.321411 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:26.322078 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:26.322229 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:26.325480 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:26.325486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:26.325504 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:26.325540 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:26.327965 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:26.328393 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:26.328399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:26.328405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:26.328516 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:26.328797 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:26.328802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:26.328803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:26.329190 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:26.329422 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:26.329663 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 22:47:26.329956 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":160,"threads_started":1}
I20260925 22:47:26.331609 12077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6065528665154b95b4290989ac791e69" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 41517 } }
I20260925 22:47:26.331943 12077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:26.332002 12077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6065528665154b95b4290989ac791e69, State: Initialized, Role: FOLLOWER
I20260925 22:47:26.332556 12077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6065528665154b95b4290989ac791e69" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 41517 } }
I20260925 22:47:26.332731 12077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:47:26.332804 12077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:47:26.332943 12077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:26.334067 12077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6065528665154b95b4290989ac791e69" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 41517 } }
I20260925 22:47:26.334419 12077 leader_election.cc:302] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6065528665154b95b4290989ac791e69; no voters:
I20260925 22:47:26.334743 12077 leader_election.cc:288] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:47:26.334847 12082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:26.335103 12082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [term 1 LEADER]: Becoming Leader. State: Replica: 6065528665154b95b4290989ac791e69, State: Running, Role: LEADER
I20260925 22:47:26.335429 12082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6065528665154b95b4290989ac791e69" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 41517 } }
I20260925 22:47:26.336079 12077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:26.337105 12084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6065528665154b95b4290989ac791e69. Latest consensus state: current_term: 1 leader_uuid: "6065528665154b95b4290989ac791e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6065528665154b95b4290989ac791e69" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 41517 } } }
I20260925 22:47:26.337221 12084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:26.337538 12083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6065528665154b95b4290989ac791e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6065528665154b95b4290989ac791e69" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 41517 } } }
I20260925 22:47:26.337625 12083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:26.338181 12077 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:47:26.338614 12091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:47:26.344275 12091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:47:26.353973 12091 catalog_manager.cc:1347] Generated new cluster ID: 2ef9fd03af104779a6cf46bbd3f090f6
I20260925 22:47:26.354126 12091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:47:26.384186 12091 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:47:26.385777 12091 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:47:26.393285 12091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69: Generated new TSK 0
I20260925 22:47:26.394213 12091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:47:27.075798 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:54275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:41517
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:27.211184 12122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:27.211511 12122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:27.211575 12122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:27.217700 12122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:27.217803 12122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:27.217931 12122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:47:27.224133 12122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.201.148:54275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:47:27.225462 12122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:27.226750 12122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:27.238111 12129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:27.238118 12132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:27.238111 12130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:27.239305 12122 server_base.cc:1034] running on GCE node
I20260925 22:47:27.239789 12122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:27.240518 12122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:27.247923 12122 hybrid_clock.cc:648] HybridClock initialized: now 1790376447247896 us; error 38 us; skew 500 ppm
Sep 25 22:47:27 dist-test-slave-vdtf krb5kdc[11565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:27.251506 12122 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:47:27.252678 12122 webserver.cc:460] Webserver started at http://127.0.201.129:44277/ using document root <none> and password file <none>
I20260925 22:47:27.253422 12122 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:27.253515 12122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:27.253772 12122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:47:27 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:47:27.270532 12122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "4fd62a80407a4805b35998c8b7fa5f8b"
format_stamp: "Formatted at 2026-09-25 22:47:27 on dist-test-slave-vdtf"
server_key: "3ef0df4b2b2a2c735a9a17cbc19a956e"
server_key_iv: "e3740c3e6ed6106357795d5abea75d0d"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:27.271195 12122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "4fd62a80407a4805b35998c8b7fa5f8b"
format_stamp: "Formatted at 2026-09-25 22:47:27 on dist-test-slave-vdtf"
server_key: "3ef0df4b2b2a2c735a9a17cbc19a956e"
server_key_iv: "e3740c3e6ed6106357795d5abea75d0d"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:27.276400 12122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 22:47:27.295632 12139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:27.297353 12122 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:47:27.297477 12122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "4fd62a80407a4805b35998c8b7fa5f8b"
format_stamp: "Formatted at 2026-09-25 22:47:27 on dist-test-slave-vdtf"
server_key: "3ef0df4b2b2a2c735a9a17cbc19a956e"
server_key_iv: "e3740c3e6ed6106357795d5abea75d0d"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:27.297598 12122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:27.313742 12122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:27.317343 12122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:27.317567 12122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:27.318192 12122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:27.319172 12122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:27.319350 12122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:27.319523 12122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:27.319617 12122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:27.331199 12122 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:44071
I20260925 22:47:27.331416 12254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:44071 every 8 connection(s)
I20260925 22:47:27.332437 12122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:47:27.334488 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 12122
I20260925 22:47:27.334612 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:47:27.334914 806 external_mini_cluster.cc:1428] Setting key 14daf5610100065970b03de1ebb0bf44
Sep 25 22:47:27 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:47:27.359400 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:27.341153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:58127 (local address 127.0.201.190:41517)
0925 22:47:27.343114 (+ 1961us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:27.343122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:27.343137 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:47:27.343179 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:27.343182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:27.343229 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:27.343329 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:27.344881 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:27.345688 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:27.350352 (+ 4664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:27.350714 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:27.350804 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:27.354536 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:27.354540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:27.354543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:27.354573 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:27.356620 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:27.357284 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:27.357301 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:27.357303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:27.357402 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:27.357893 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:27.357897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:27.357901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:27.358152 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:27.358262 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:27.358965 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 22:47:27.359106 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1888,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:47:27.362089 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:27.340060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:41517 (local address 127.0.201.129:58127)
0925 22:47:27.341612 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:27.341671 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:27.342700 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:27.343635 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:27.343644 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:27.344015 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:27.344700 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:27.344723 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:27.348786 (+ 4063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:27.348790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:27.350189 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:27.350202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:27.351012 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:27.351015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:27.351261 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:27.352089 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:27.352111 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:27.354328 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:27.356769 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:27.356775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:27.356792 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:27.357160 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:27.357533 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:27.357536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:27.357539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:27.357733 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:27.358305 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:27.358312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:27.358650 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 22:47:27.358861 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 22:47:27.359205 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1323,"thread_start_us":155,"threads_started":1}
I20260925 22:47:27.364178 12255 heartbeater.cc:347] Connected to a master server at 127.0.201.190:41517
I20260925 22:47:27.364447 12255 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:27.365078 12255 heartbeater.cc:511] Master 127.0.201.190:41517 requested a full tablet report, sending...
I20260925 22:47:27.366956 12041 ts_manager.cc:194] Registered new tserver with Master: 4fd62a80407a4805b35998c8b7fa5f8b (127.0.201.129:44071)
I20260925 22:47:27.368666 12041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:58127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:47:27.379135 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:54275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:41517
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:47:27 dist-test-slave-vdtf krb5kdc[11565](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 22:47:27.530824 12261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:27.531101 12261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:27.531201 12261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:27.535411 12261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:27.535488 12261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:27.535586 12261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:47:27.541364 12261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.201.148:54275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:47:27.542424 12261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:27.543519 12261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:27.555040 12267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:27.555282 12261 server_base.cc:1034] running on GCE node
W20260925 22:47:27.555550 12270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:27.555783 12268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:27.556237 12261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:27.557032 12261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:27.558229 12261 hybrid_clock.cc:648] HybridClock initialized: now 1790376447558194 us; error 49 us; skew 500 ppm
Sep 25 22:47:27 dist-test-slave-vdtf krb5kdc[11565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:27.561935 12261 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:47:27.562891 12261 webserver.cc:460] Webserver started at http://127.0.201.130:42225/ using document root <none> and password file <none>
I20260925 22:47:27.563450 12261 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:27.563539 12261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:27.563827 12261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:47:27 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:47:27.577865 12261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "a9c5714f2eb345218be3792d4980d35e"
format_stamp: "Formatted at 2026-09-25 22:47:27 on dist-test-slave-vdtf"
server_key: "bdcf47b7ab885687cd0454f4c25e3d5c"
server_key_iv: "043b3a06a692144dad744bd10404ee96"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:27.578532 12261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "a9c5714f2eb345218be3792d4980d35e"
format_stamp: "Formatted at 2026-09-25 22:47:27 on dist-test-slave-vdtf"
server_key: "bdcf47b7ab885687cd0454f4c25e3d5c"
server_key_iv: "043b3a06a692144dad744bd10404ee96"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:27.582484 12261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:47:27.600067 12277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:27.601830 12261 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:47:27.601961 12261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "a9c5714f2eb345218be3792d4980d35e"
format_stamp: "Formatted at 2026-09-25 22:47:27 on dist-test-slave-vdtf"
server_key: "bdcf47b7ab885687cd0454f4c25e3d5c"
server_key_iv: "043b3a06a692144dad744bd10404ee96"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:27.602092 12261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:27.637770 12261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:27.646696 12261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:27.646986 12261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:27.647828 12261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:27.648947 12261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:27.649046 12261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:27.649139 12261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:27.649199 12261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:27.661548 12261 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:43637
I20260925 22:47:27.663791 12261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:47:27.666368 12390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:43637 every 8 connection(s)
I20260925 22:47:27.669093 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 12261
I20260925 22:47:27.669219 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:47:27.669535 806 external_mini_cluster.cc:1428] Setting key 97e56d9d81a27cade72e7edee8741776
Sep 25 22:47:27 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:47:27.684098 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:27.669912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:35685 (local address 127.0.201.190:41517)
0925 22:47:27.670077 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:27.670082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:27.670679 (+ 597us) server_negotiation.cc:407] Connection header received
0925 22:47:27.671932 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:27.671939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:27.672000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:27.672056 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:27.673710 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:27.674539 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:27.676256 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:27.676603 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:27.676680 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:27.679751 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:27.679757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:27.679762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:27.679795 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:27.681657 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:27.682272 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:27.682276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:27.682277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:27.682364 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:27.682763 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:27.682767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:27.682769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:27.683012 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:27.683088 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:27.683791 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 22:47:27.683928 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:47:27.686094 12393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:27.668731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:41517 (local address 127.0.201.130:35685)
0925 22:47:27.670526 (+ 1795us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:27.670584 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:27.671663 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:27.672490 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:27.672502 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:27.672880 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:27.673544 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:27.673564 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:27.674696 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:27.674699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:27.676119 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:27.676130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:27.676882 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:27.676886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:27.677080 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:27.677754 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:27.677775 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:27.679529 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:27.681785 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:27.681790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:27.681819 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:27.682152 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:27.682477 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:27.682480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:27.682482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:27.682651 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:27.683173 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:27.683179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:27.683481 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 22:47:27.684571 (+ 1090us) client_negotiation.cc:240] Negotiation successful
0925 22:47:27.684937 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1578,"thread_start_us":150,"threads_started":1}
I20260925 22:47:27.687701 12391 heartbeater.cc:347] Connected to a master server at 127.0.201.190:41517
I20260925 22:47:27.687956 12391 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:27.688496 12391 heartbeater.cc:511] Master 127.0.201.190:41517 requested a full tablet report, sending...
I20260925 22:47:27.689612 12041 ts_manager.cc:194] Registered new tserver with Master: a9c5714f2eb345218be3792d4980d35e (127.0.201.130:43637)
I20260925 22:47:27.690356 12041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:35685
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:47:27.729089 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:54275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:41517
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:27.900156 12396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:27.900507 12396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:27.900588 12396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:27.906589 12396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:27.906725 12396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:27.906845 12396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:47:27.914585 12396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.201.148:54275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:47:27.916194 12396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:27.917574 12396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:27.930042 12404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:27.930042 12401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:27.930065 12402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:27.931838 12396 server_base.cc:1034] running on GCE node
I20260925 22:47:27.932444 12396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:27.933192 12396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:27.936491 12396 hybrid_clock.cc:648] HybridClock initialized: now 1790376447936421 us; error 83 us; skew 500 ppm
Sep 25 22:47:27 dist-test-slave-vdtf krb5kdc[11565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:27.940526 12396 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:47:27.941625 12396 webserver.cc:460] Webserver started at http://127.0.201.131:44483/ using document root <none> and password file <none>
I20260925 22:47:27.942313 12396 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:27.942381 12396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:27.942586 12396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:47:27 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
Sep 25 22:47:27 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:47:27.966547 12396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "11c470b87d7541e983a15792d7b1a2ed"
format_stamp: "Formatted at 2026-09-25 22:47:27 on dist-test-slave-vdtf"
server_key: "c0fbe45838a00f4c7efe97d65f198f82"
server_key_iv: "2b19de19b268cbcee0579098855667a8"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:27.967638 12396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "11c470b87d7541e983a15792d7b1a2ed"
format_stamp: "Formatted at 2026-09-25 22:47:27 on dist-test-slave-vdtf"
server_key: "c0fbe45838a00f4c7efe97d65f198f82"
server_key_iv: "2b19de19b268cbcee0579098855667a8"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:27.987365 12396 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.004s sys 0.000s
I20260925 22:47:28.007108 12411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:28.008790 12396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:47:28.008903 12396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "11c470b87d7541e983a15792d7b1a2ed"
format_stamp: "Formatted at 2026-09-25 22:47:27 on dist-test-slave-vdtf"
server_key: "c0fbe45838a00f4c7efe97d65f198f82"
server_key_iv: "2b19de19b268cbcee0579098855667a8"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:28.009019 12396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:28.048857 12396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:28.052475 12396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:28.052696 12396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:28.053371 12396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:28.054409 12396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:28.054507 12396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:28.054594 12396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:28.054651 12396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:28.064846 12396 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:43749
I20260925 22:47:28.066005 12396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:47:28.069326 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 12396
I20260925 22:47:28.069438 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:47:28.069765 806 external_mini_cluster.cc:1428] Setting key ead1ce72128a256654d4bdfc7533a5a8
I20260925 22:47:28.077376 12524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:43749 every 8 connection(s)
Sep 25 22:47:28 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:47:28.091358 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.079519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:57603 (local address 127.0.201.190:41517)
0925 22:47:28.079628 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:28.079632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:28.079644 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:47:28.080350 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:28.080355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:28.080407 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:28.080503 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:28.081945 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.082698 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.084427 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.084748 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.084876 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.087539 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:28.087543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:28.087545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:28.087573 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:28.089172 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:28.089669 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:28.089672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:28.089674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:28.089760 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:28.090199 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:28.090202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:28.090204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:28.090434 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:28.090534 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:28.091084 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 22:47:28.091204 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 22:47:28.092209 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.078439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:41517 (local address 127.0.201.131:57603)
0925 22:47:28.079004 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:28.079065 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:28.080086 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:28.080708 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:28.080717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:28.081080 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:28.081769 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.081789 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.082843 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.082846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.084287 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.084298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.084943 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.084946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.085095 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.085814 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:28.085835 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:28.087359 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:28.089293 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:28.089298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:28.089311 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:28.089563 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:28.089922 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:28.089924 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:28.089926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:28.090094 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:28.090574 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:28.090580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:28.090837 (+ 257us) client_negotiation.cc:769] Sending connection context
0925 22:47:28.091009 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 22:47:28.091258 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":150,"threads_started":1}
I20260925 22:47:28.093386 12525 heartbeater.cc:347] Connected to a master server at 127.0.201.190:41517
I20260925 22:47:28.093650 12525 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:28.094285 12525 heartbeater.cc:511] Master 127.0.201.190:41517 requested a full tablet report, sending...
I20260925 22:47:28.095425 12041 ts_manager.cc:194] Registered new tserver with Master: 11c470b87d7541e983a15792d7b1a2ed (127.0.201.131:43749)
I20260925 22:47:28.096088 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:47:28.096215 12041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:57603
I20260925 22:47:28.106717 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.099218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.0.201.190:41517)
0925 22:47:28.099339 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:28.099343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:28.099551 (+ 208us) server_negotiation.cc:407] Connection header received
0925 22:47:28.099720 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:28.099723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:28.099771 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:28.099856 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:28.100547 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.101063 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.101903 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.102125 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.102228 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.103713 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:28.103718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:28.103721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:28.103745 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:28.105130 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:28.105550 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:28.105557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:28.105559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:28.105637 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:28.105909 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:28.105912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:28.105914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:28.106077 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:28.106210 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:28.106445 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 22:47:28.106553 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:47:28.110059 12041 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:47:28.113613 12041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:47:28.134272 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.123241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44071 (local address 127.0.0.1:44820)
0925 22:47:28.127924 (+ 4683us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:28.127961 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:28.128120 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:28.129083 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:28.129089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:28.129121 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:28.129444 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.129451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.130985 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.130989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.132446 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.132455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.133721 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.133724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.133813 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.133867 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 22:47:28.133970 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 22:47:28.134094 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4549,"mutex_wait_us":108,"spinlock_wait_cycles":11904,"thread_start_us":106,"threads_started":1}
I20260925 22:47:28.134883 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.123430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.0.201.129:44071)
0925 22:47:28.123778 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:28.123786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:28.128060 (+ 4274us) server_negotiation.cc:407] Connection header received
0925 22:47:28.128285 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:28.128291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:28.128761 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:28.128904 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:28.129570 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.130859 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.132579 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.133396 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.133500 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.133670 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:28.134645 (+ 975us) server_negotiation.cc:299] Negotiation successful
0925 22:47:28.134754 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
I20260925 22:47:28.134702 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.123654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:43749 (local address 127.0.0.1:44836)
0925 22:47:28.128342 (+ 4688us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:28.128360 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:28.128442 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:28.129091 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:28.129094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:28.129121 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:28.129443 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.129451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.131464 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.131468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.132607 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.132614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.133624 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.133627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.134436 (+ 809us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.134459 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:47:28.134508 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 22:47:28.134572 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4635,"mutex_wait_us":85,"spinlock_wait_cycles":4352,"thread_start_us":88,"threads_started":1}
I20260925 22:47:28.135565 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.124411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.0.201.131:43749)
0925 22:47:28.128675 (+ 4264us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:28.128687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:28.128707 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:47:28.128776 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:28.128780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:28.128959 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:28.129579 (+ 620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:28.129711 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.131341 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.132758 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.133501 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.134798 (+ 1297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.134959 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:28.135014 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 22:47:28.135101 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4167,"thread_start_us":121,"threads_started":1}
I20260925 22:47:28.138324 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.122685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:43637 (local address 127.0.0.1:58784)
0925 22:47:28.127926 (+ 5241us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:28.127965 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:28.128120 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:28.132343 (+ 4223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:28.132346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:28.132363 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:28.132464 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.132470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.134295 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.134298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.136376 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.136386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.137850 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.137853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.137958 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.137979 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:47:28.138044 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 22:47:28.138109 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5068,"spinlock_wait_cycles":8192,"thread_start_us":93,"threads_started":1}
I20260925 22:47:28.138367 12172 tablet_service.cc:1467] Processing CreateTablet for tablet 4d940f5b72f3489a8ed5e6cc96bfab86 (DEFAULT_TABLE table=test-table [id=37efe098db7946c396827038198df167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:28.138368 12449 tablet_service.cc:1467] Processing CreateTablet for tablet 4d940f5b72f3489a8ed5e6cc96bfab86 (DEFAULT_TABLE table=test-table [id=37efe098db7946c396827038198df167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:28.139488 12172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d940f5b72f3489a8ed5e6cc96bfab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:28.139488 12449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d940f5b72f3489a8ed5e6cc96bfab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:28.148039 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.123954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.0.201.130:43637)
0925 22:47:28.124197 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:28.124213 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:28.131743 (+ 7530us) server_negotiation.cc:407] Connection header received
0925 22:47:28.131829 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:28.131834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:28.132015 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:28.132165 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:28.132605 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.134152 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.136530 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.137572 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.137669 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.137794 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:28.147766 (+ 9972us) server_negotiation.cc:299] Negotiation successful
0925 22:47:28.147894 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260925 22:47:28.148569 12544 tablet_bootstrap.cc:492] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: Bootstrap starting.
I20260925 22:47:28.151213 12544 tablet_bootstrap.cc:654] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:28.151314 12324 tablet_service.cc:1467] Processing CreateTablet for tablet 4d940f5b72f3489a8ed5e6cc96bfab86 (DEFAULT_TABLE table=test-table [id=37efe098db7946c396827038198df167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:28.152119 12544 log.cc:824] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:28.152447 12324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d940f5b72f3489a8ed5e6cc96bfab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:28.152786 12543 tablet_bootstrap.cc:492] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b: Bootstrap starting.
I20260925 22:47:28.155228 12543 tablet_bootstrap.cc:654] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:28.155983 12543 log.cc:824] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:28.160043 12547 tablet_bootstrap.cc:492] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: Bootstrap starting.
I20260925 22:47:28.162115 12547 tablet_bootstrap.cc:654] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:28.162855 12543 tablet_bootstrap.cc:492] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b: No bootstrap required, opened a new log
I20260925 22:47:28.163131 12543 ts_tablet_manager.cc:1397] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260925 22:47:28.163482 12547 log.cc:824] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:28.165864 12544 tablet_bootstrap.cc:492] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: No bootstrap required, opened a new log
I20260925 22:47:28.166052 12544 ts_tablet_manager.cc:1397] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260925 22:47:28.168983 12543 raft_consensus.cc:348] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 } } peers { permanent_uuid: "11c470b87d7541e983a15792d7b1a2ed" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 43749 } } peers { permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 } }
I20260925 22:47:28.169283 12543 raft_consensus.cc:374] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:28.169371 12543 raft_consensus.cc:729] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd62a80407a4805b35998c8b7fa5f8b, State: Initialized, Role: FOLLOWER
I20260925 22:47:28.169862 12543 consensus_queue.cc:260] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 } } peers { permanent_uuid: "11c470b87d7541e983a15792d7b1a2ed" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 43749 } } peers { permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 } }
I20260925 22:47:28.170325 12544 raft_consensus.cc:348] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 } } peers { permanent_uuid: "11c470b87d7541e983a15792d7b1a2ed" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 43749 } } peers { permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 } }
I20260925 22:47:28.170768 12544 raft_consensus.cc:374] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:28.170878 12544 raft_consensus.cc:729] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c470b87d7541e983a15792d7b1a2ed, State: Initialized, Role: FOLLOWER
I20260925 22:47:28.170904 12543 ts_tablet_manager.cc:1428] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:47:28.171322 12255 heartbeater.cc:503] Master 127.0.201.190:41517 was elected leader, sending a full tablet report...
I20260925 22:47:28.171379 12544 consensus_queue.cc:260] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 } } peers { permanent_uuid: "11c470b87d7541e983a15792d7b1a2ed" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 43749 } } peers { permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 } }
I20260925 22:47:28.172314 12544 ts_tablet_manager.cc:1428] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:47:28.172631 12525 heartbeater.cc:503] Master 127.0.201.190:41517 was elected leader, sending a full tablet report...
I20260925 22:47:28.173033 12547 tablet_bootstrap.cc:492] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: No bootstrap required, opened a new log
I20260925 22:47:28.173202 12547 ts_tablet_manager.cc:1397] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260925 22:47:28.177579 12547 raft_consensus.cc:348] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 } } peers { permanent_uuid: "11c470b87d7541e983a15792d7b1a2ed" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 43749 } } peers { permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 } }
I20260925 22:47:28.177810 12547 raft_consensus.cc:374] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:28.177881 12547 raft_consensus.cc:729] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c5714f2eb345218be3792d4980d35e, State: Initialized, Role: FOLLOWER
I20260925 22:47:28.178268 12547 consensus_queue.cc:260] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 } } peers { permanent_uuid: "11c470b87d7541e983a15792d7b1a2ed" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 43749 } } peers { permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 } }
I20260925 22:47:28.179136 12547 ts_tablet_manager.cc:1428] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:47:28.179462 12391 heartbeater.cc:503] Master 127.0.201.190:41517 was elected leader, sending a full tablet report...
I20260925 22:47:28.213671 12550 raft_consensus.cc:482] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:28.213959 12550 raft_consensus.cc:504] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 } } peers { permanent_uuid: "11c470b87d7541e983a15792d7b1a2ed" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 43749 } } peers { permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 } }
I20260925 22:47:28.216310 12550 leader_election.cc:288] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9c5714f2eb345218be3792d4980d35e (127.0.201.130:43637), 4fd62a80407a4805b35998c8b7fa5f8b (127.0.201.129:44071)
I20260925 22:47:28.221956 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.216988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:32943 (local address 127.0.201.129:44071)
0925 22:47:28.217115 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:28.217119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:28.217450 (+ 331us) server_negotiation.cc:407] Connection header received
0925 22:47:28.217545 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:28.217548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:28.217593 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:28.217669 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:28.217992 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.218769 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.220563 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.221189 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.221285 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.221351 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:28.221736 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 22:47:28.221817 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:47:28.221982 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.215651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:43637 (local address 127.0.201.131:33029)
0925 22:47:28.215965 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:28.215981 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:28.216082 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:28.216924 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:28.216928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:28.216950 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:28.217071 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.217078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.219250 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.219253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.220564 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.220575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.221538 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.221542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.221681 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.221702 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:47:28.221769 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 22:47:28.221847 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":29}
I20260925 22:47:28.222442 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.216092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:33029 (local address 127.0.201.130:43637)
0925 22:47:28.216223 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:28.216227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:28.216239 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:47:28.216290 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:28.216293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:28.216369 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:28.216449 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:28.217193 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.217881 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.220711 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.221319 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.221434 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.221481 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:28.222246 (+ 765us) server_negotiation.cc:299] Negotiation successful
0925 22:47:28.222329 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:47:28.222988 12329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d940f5b72f3489a8ed5e6cc96bfab86" candidate_uuid: "11c470b87d7541e983a15792d7b1a2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9c5714f2eb345218be3792d4980d35e" is_pre_election: true
I20260925 22:47:28.223310 12329 raft_consensus.cc:2454] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c470b87d7541e983a15792d7b1a2ed in term 0.
I20260925 22:47:28.224002 12414 leader_election.cc:302] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c470b87d7541e983a15792d7b1a2ed, a9c5714f2eb345218be3792d4980d35e; no voters:
I20260925 22:47:28.224301 12550 raft_consensus.cc:2781] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:47:28.224395 12550 raft_consensus.cc:482] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:47:28.224458 12550 raft_consensus.cc:3037] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:28.224651 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.216526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44071 (local address 127.0.201.131:32943)
0925 22:47:28.217331 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:28.217347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:28.217460 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:28.217707 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:28.217709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:28.217727 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:28.217849 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.217856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.218921 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.218924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.220416 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:28.220427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.221462 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.221465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:28.221590 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.221614 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:47:28.222439 (+ 825us) client_negotiation.cc:240] Negotiation successful
0925 22:47:28.224361 (+ 1922us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":30,"thread_start_us":123,"threads_started":1}
I20260925 22:47:28.225179 12193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d940f5b72f3489a8ed5e6cc96bfab86" candidate_uuid: "11c470b87d7541e983a15792d7b1a2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" is_pre_election: true
I20260925 22:47:28.225458 12193 raft_consensus.cc:2454] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c470b87d7541e983a15792d7b1a2ed in term 0.
I20260925 22:47:28.225832 12550 raft_consensus.cc:504] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 } } peers { permanent_uuid: "11c470b87d7541e983a15792d7b1a2ed" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 43749 } } peers { permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 } }
I20260925 22:47:28.226197 12550 leader_election.cc:288] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [CANDIDATE]: Term 1 election: Requested vote from peers a9c5714f2eb345218be3792d4980d35e (127.0.201.130:43637), 4fd62a80407a4805b35998c8b7fa5f8b (127.0.201.129:44071)
I20260925 22:47:28.226630 12329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d940f5b72f3489a8ed5e6cc96bfab86" candidate_uuid: "11c470b87d7541e983a15792d7b1a2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9c5714f2eb345218be3792d4980d35e"
I20260925 22:47:28.226715 12193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d940f5b72f3489a8ed5e6cc96bfab86" candidate_uuid: "11c470b87d7541e983a15792d7b1a2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd62a80407a4805b35998c8b7fa5f8b"
I20260925 22:47:28.226753 12329 raft_consensus.cc:3037] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:28.226818 12193 raft_consensus.cc:3037] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:28.228133 12193 raft_consensus.cc:2454] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c470b87d7541e983a15792d7b1a2ed in term 1.
I20260925 22:47:28.228152 12329 raft_consensus.cc:2454] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c470b87d7541e983a15792d7b1a2ed in term 1.
I20260925 22:47:28.228477 12414 leader_election.cc:302] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c470b87d7541e983a15792d7b1a2ed, a9c5714f2eb345218be3792d4980d35e; no voters:
I20260925 22:47:28.228705 12550 raft_consensus.cc:2781] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:28.228925 12550 raft_consensus.cc:686] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 1 LEADER]: Becoming Leader. State: Replica: 11c470b87d7541e983a15792d7b1a2ed, State: Running, Role: LEADER
I20260925 22:47:28.229264 12550 consensus_queue.cc:237] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 } } peers { permanent_uuid: "11c470b87d7541e983a15792d7b1a2ed" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 43749 } } peers { permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 } }
I20260925 22:47:28.232589 12041 catalog_manager.cc:5495] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 11c470b87d7541e983a15792d7b1a2ed (127.0.201.131). New cstate: current_term: 1 leader_uuid: "11c470b87d7541e983a15792d7b1a2ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c470b87d7541e983a15792d7b1a2ed" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 43749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 } health_report { overall_health: UNKNOWN } } }
I20260925 22:47:28.261970 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.256580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.0.201.131:43749)
0925 22:47:28.256991 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:28.256996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:28.257010 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:47:28.257094 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:28.257097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:28.257163 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:28.257265 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:47:28.257660 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.258469 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.259732 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:28.260062 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:28.260167 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:28.260617 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:47:28.261195 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:47:28.261329 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:28.261747 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 22:47:28.261807 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325}
I20260925 22:47:28.269073 12193 raft_consensus.cc:1261] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 4fd62a80407a4805b35998c8b7fa5f8b [term 1 FOLLOWER]: Refusing update from remote peer 11c470b87d7541e983a15792d7b1a2ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:28.269194 12329 raft_consensus.cc:1261] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e [term 1 FOLLOWER]: Refusing update from remote peer 11c470b87d7541e983a15792d7b1a2ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:28.269939 12550 consensus_queue.cc:1035] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c5714f2eb345218be3792d4980d35e" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:28.270136 12553 consensus_queue.cc:1035] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fd62a80407a4805b35998c8b7fa5f8b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:28.284953 12556 mvcc.cc:205] Tried to move back new op lower bound from 7333381932101877760 to 7333381931950108672. Current Snapshot: MvccSnapshot[applied={T|T < 7333381932101877760}]
Sep 25 22:47:28 dist-test-slave-vdtf krb5kdc[11565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:47:28.292006 12559 mvcc.cc:205] Tried to move back new op lower bound from 7333381932101877760 to 7333381931950108672. Current Snapshot: MvccSnapshot[applied={T|T < 7333381932101877760 or (T in {7333381932101877760})}]
I20260925 22:47:28.296706 12040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:28.296912 12040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:28.299587 12040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6065528665154b95b4290989ac791e69: Sending DeleteTablet for 3 replicas of tablet 4d940f5b72f3489a8ed5e6cc96bfab86
I20260925 22:47:28.300390 12449 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d940f5b72f3489a8ed5e6cc96bfab86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:28 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:44836
I20260925 22:47:28.300518 12324 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d940f5b72f3489a8ed5e6cc96bfab86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:28 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:58784
I20260925 22:47:28.301415 12172 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d940f5b72f3489a8ed5e6cc96bfab86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:28 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:44820
I20260925 22:47:28.304634 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 12122
I20260925 22:47:28.305423 12566 tablet_replica.cc:330] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: stopping tablet replica
I20260925 22:47:28.305778 12566 raft_consensus.cc:2229] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:28.306041 12566 raft_consensus.cc:2258] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:28.311352 12565 tablet_replica.cc:330] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: stopping tablet replica
I20260925 22:47:28.315996 12565 raft_consensus.cc:2229] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:47:28.316491 12565 raft_consensus.cc:2258] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:28.318027 12565 ts_tablet_manager.cc:1905] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:28.321600 12566 ts_tablet_manager.cc:1905] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:28.324337 12566 ts_tablet_manager.cc:1918] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:28.324530 12566 log.cc:1196] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/4d940f5b72f3489a8ed5e6cc96bfab86
I20260925 22:47:28.324937 12566 ts_tablet_manager.cc:1939] T 4d940f5b72f3489a8ed5e6cc96bfab86 P a9c5714f2eb345218be3792d4980d35e: Deleting consensus metadata
I20260925 22:47:28.326337 12027 catalog_manager.cc:4841] TS a9c5714f2eb345218be3792d4980d35e (127.0.201.130:43637): tablet 4d940f5b72f3489a8ed5e6cc96bfab86 (table test-table [id=37efe098db7946c396827038198df167]) successfully deleted
I20260925 22:47:28.327533 12565 ts_tablet_manager.cc:1918] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:28.328764 12565 log.cc:1196] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/4d940f5b72f3489a8ed5e6cc96bfab86
I20260925 22:47:28.329386 12565 ts_tablet_manager.cc:1939] T 4d940f5b72f3489a8ed5e6cc96bfab86 P 11c470b87d7541e983a15792d7b1a2ed: Deleting consensus metadata
W20260925 22:47:28.329654 12026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:44071 (error 108)
I20260925 22:47:28.331660 12028 catalog_manager.cc:4841] TS 11c470b87d7541e983a15792d7b1a2ed (127.0.201.131:43749): tablet 4d940f5b72f3489a8ed5e6cc96bfab86 (table test-table [id=37efe098db7946c396827038198df167]) successfully deleted
I20260925 22:47:28.332163 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 12261
I20260925 22:47:28.337342 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.336990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44071 (local address 127.0.0.1:44832)
0925 22:47:28.337142 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:28.337229 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 22:47:28.337574 12026 catalog_manager.cc:4584] TS 4fd62a80407a4805b35998c8b7fa5f8b (127.0.201.129:44071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d940f5b72f3489a8ed5e6cc96bfab86: Network error: Client connection negotiation failed: client connection to 127.0.201.129:44071: connect: Connection refused (error 111)
I20260925 22:47:28.339942 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 12396
I20260925 22:47:28.346650 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 12009
I20260925 22:47:28.355036 806 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:47:28.700073 806 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:47:28.700193 806 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:47:29.044759 806 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:47:29.044 UTC [11595] LOG: received smart shutdown request
2026-09-25 22:47:29.048 UTC [11595] LOG: background worker "logical replication launcher" (PID 11604) exited with exit code 1
2026-09-25 22:47:29.048 UTC [11599] LOG: shutting down
2026-09-25 22:47:29.216 UTC [11595] LOG: database system is shut down
2026-09-25T22:47:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45503 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:47:29 dist-test-slave-vdtf krb5kdc[12579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:47:29 dist-test-slave-vdtf krb5kdc[12579](info): set up 2 sockets
Sep 25 22:47:29 dist-test-slave-vdtf krb5kdc[12579](info): commencing operation
krb5kdc: starting...
W20260925 22:47:30.399863 806 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:47:30 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:47:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.201.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.201.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/rangeradmin_127.0.201.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/rangeradmin_127.0.201.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.201.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.201.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/rangerlookup_127.0.201.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/rangerlookup_127.0.201.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.201.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.201.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/HTTP_127.0.201.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/HTTP_127.0.201.148@KRBTEST.COM.keytab.
I20260925 22:47:30.566692 806 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrAtchr/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/postgres -l logfile start
2026-09-25 22:47:32.975 UTC [12609] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:47:32.975 UTC [12609] LOG: listening on IPv4 address "127.0.201.148", port 41925
2026-09-25 22:47:32.979 UTC [12609] LOG: listening on Unix socket "/tmp/.s.PGSQL.41925"
2026-09-25 22:47:33.013 UTC [12612] LOG: database system was shut down at 2026-09-25 22:47:31 UTC
2026-09-25 22:47:33.017 UTC [12609] LOG: database system is ready to accept connections
I20260925 22:47:33.959555 806 mini_postgres.cc:96] Postgres bound to 41925
2026-09-25 22:47:33.965 UTC [12620] FATAL: database "slave" does not exist
127.0.201.148:41925 - accepting connections
I20260925 22:47:33.966259 806 mini_ranger.cc:161] Starting Ranger...
I20260925 22:47:33.980772 806 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:47:34.437711 806 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:47:34.437776 806 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-admin
2026-09-25 22:47:34,690 [I] DB FLAVOR :POSTGRES
2026-09-25 22:47:34,691 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:47:34,691 [I] Checking connection..
2026-09-25 22:47:34,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:47:34,926 [I] Checking connection passed.
2026-09-25 22:47:34,926 [I] --------- Verifying version history table ---------
2026-09-25 22:47:34,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:47:35,152 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:47:35,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:47:35,407 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:47:35,407 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:47:35,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:47:35.627 UTC [12684] WARNING: there is no transaction in progress
2026-09-25 22:47:35,633 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:47:35,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:47:35,859 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:47:35,859 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:47:35,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:47:36,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:47:36,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vdtf.c.gcp-upstream.internal','N') ;"
2026-09-25 22:47:36,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:47:36,822 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:47:36,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:47:37,073 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:47:37,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:47:37,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:47:37,548 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:47:37,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:47:38.508 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.524 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.540 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.554 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.570 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.590 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.654 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.661 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.674 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.685 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.694 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.707 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.716 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.724 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.730 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:38.737 UTC [12792] WARNING: there is no transaction in progress
2026-09-25 22:47:39,345 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:47:39,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:47:39,564 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:47:39,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:47:39,818 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:47:39,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:47:40,045 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:47:40,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:47:40,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:47:40,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vdtf.c.gcp-upstream.internal';"
2026-09-25 22:47:40,781 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:47:40,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:47:41,015 [I] DB_PATCHES have already been applied
I20260925 22:47:41.021119 806 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrAtchr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:47:41.021180 806 mini_ranger.cc:193] Using host: 127.0.201.148
I20260925 22:47:41.024536 806 mini_ranger.cc:227] Ranger admin URL: http://127.0.201.148:53831
Sep 25, 2026 10:47:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:47:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:47:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:47:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:47:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:47:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:47:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:47:42.196719 806 mini_ranger.cc:160] Time spent starting Ranger: real 8.230s user 0.000s sys 0.009s
Sep 25, 2026 10:48:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:48:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 10:48:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:48:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:48:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:48:04.209110 806 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.201.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.201.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/rangerkms_127.0.201.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.201.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/rangerkms_127.0.201.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.201.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.201.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.201.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/HTTP_127.0.201.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.201.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/HTTP_127.0.201.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:48:04 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376484, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:48:04.297775 806 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:48:04.307039 806 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:48:05.556718 806 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:48:05.556833 806 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-kms
I20260925 22:48:05.556857 806 mini_ranger_kms.cc:227] Using postgres at 127.0.201.148:41925
2026-09-25 22:48:05,723 [I] DB FLAVOR :POSTGRES
2026-09-25 22:48:05,723 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:48:05,723 [I] Checking connection
2026-09-25 22:48:05,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:48:05,961 [I] connection success
2026-09-25 22:48:05,961 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:48:05,961 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:48:05,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:48:06,222 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:48:06,222 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:48:06,223 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:48:06,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.201.148:41925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:48:06,499 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:48:06.968223 806 mini_ranger_kms.cc:310] Created kms service
I20260925 22:48:07.135283 806 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:48:07.176854 806 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:48:07 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376484, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:07.557116 806 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:48:07.557267 806 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrAtchr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrAtchr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:48:07.557289 806 mini_ranger_kms.cc:240] Using host: 127.0.201.148
I20260925 22:48:07.561244 806 mini_ranger_kms.cc:276] Ranger KMS PID: 12972
I20260925 22:48:07.561326 806 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.201.148:57221
Sep 25, 2026 10:48:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:48:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:48:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:48:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:48:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:48:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 22:48:09 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376489, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.201.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:09.822115 806 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.524s user 0.000s sys 0.150s
I20260925 22:48:09.822216 806 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:48:09.822266 806 mini_ranger_kms.cc:401] 127.0.201.148:57221/kms/v1/keys
Sep 25 22:48:10 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376489, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.201.148@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
Sep 25 22:48:10 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376489, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.201.148@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
Sep 25 22:48:12 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:48:12.611114 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:33675
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:33675
--ranger_config_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:57221/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:12.733470 13023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:12.733707 13023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:12.733744 13023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:12.737766 13023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:12.737819 13023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:12.737838 13023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:12.737854 13023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:12.737891 13023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:12.743144 13023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.201.148:57221/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:33675
--ranger_config_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:33675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:12.744163 13023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:12.745121 13023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:12.753347 13031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:12.753345 13028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:12.753409 13023 server_base.cc:1034] running on GCE node
W20260925 22:48:12.753347 13029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:12.754235 13023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:12.755267 13023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:12.756451 13023 hybrid_clock.cc:648] HybridClock initialized: now 1790376492756432 us; error 37 us; skew 500 ppm
Sep 25 22:48:12 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:12.759335 13023 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:48:12.760309 13023 webserver.cc:460] Webserver started at http://127.0.201.190:44743/ using document root <none> and password file <none>
I20260925 22:48:12.760831 13023 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:12.760886 13023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:12.761093 13023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:48:12 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:12.807054 13023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "8ff36b9e4d624ea5b6b38443ef7477f7"
format_stamp: "Formatted at 2026-09-25 22:48:12 on dist-test-slave-vdtf"
server_key: "b033e7416e4d0048609a4a5a75705c41"
server_key_iv: "aef3e2f1ac9ec78b74bb33f7ba818673"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:12.807744 13023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "8ff36b9e4d624ea5b6b38443ef7477f7"
format_stamp: "Formatted at 2026-09-25 22:48:12 on dist-test-slave-vdtf"
server_key: "b033e7416e4d0048609a4a5a75705c41"
server_key_iv: "aef3e2f1ac9ec78b74bb33f7ba818673"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:12.811734 13023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 22:48:12.825917 13038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:12.827370 13023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 22:48:12.827478 13023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "8ff36b9e4d624ea5b6b38443ef7477f7"
format_stamp: "Formatted at 2026-09-25 22:48:12 on dist-test-slave-vdtf"
server_key: "b033e7416e4d0048609a4a5a75705c41"
server_key_iv: "aef3e2f1ac9ec78b74bb33f7ba818673"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:12.827633 13023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:12.837913 13023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:12.840781 13023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:12.840974 13023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:12.849074 13023 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:33675
I20260925 22:48:12.849080 13090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:33675 every 8 connection(s)
I20260925 22:48:12.850044 13023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:12.852857 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:12.858225 13091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: Bootstrap starting.
I20260925 22:48:12.858843 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 13023
I20260925 22:48:12.859004 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:48:12.859323 806 external_mini_cluster.cc:1428] Setting key 9a19cd6b44672a624ab060705f5a766b
I20260925 22:48:12.861076 13091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:12.861961 13091 log.cc:824] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:12.864283 13091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: No bootstrap required, opened a new log
Sep 25 22:48:12 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:12.869035 13091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } }
I20260925 22:48:12.869381 13091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:12.869447 13091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff36b9e4d624ea5b6b38443ef7477f7, State: Initialized, Role: FOLLOWER
I20260925 22:48:12.869967 13091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } }
I20260925 22:48:12.870121 13091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:12.870206 13091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:12.870326 13091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:12.871443 13091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } }
I20260925 22:48:12.871855 13091 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff36b9e4d624ea5b6b38443ef7477f7; no voters:
I20260925 22:48:12.872226 13091 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:12.873006 13097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:12.873476 13097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 1 LEADER]: Becoming Leader. State: Replica: 8ff36b9e4d624ea5b6b38443ef7477f7, State: Running, Role: LEADER
I20260925 22:48:12.873594 13091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:12.873852 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:12.860561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.0.201.190:33675)
0925 22:48:12.861052 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:12.861072 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:12.861114 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:48:12.861980 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:12.862015 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:12.862441 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:12.862779 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:12.863570 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:12.864702 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:12.865551 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:12.866036 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:12.866197 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:12.868726 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:12.868731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:12.868745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:12.868771 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:12.870684 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:12.871126 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:12.871132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:12.871137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:12.871242 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:12.871524 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:12.871529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:12.871530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:12.872041 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:12.872253 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:12.872569 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 22:48:12.872872 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":153,"threads_started":1}
I20260925 22:48:12.874102 13097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } }
I20260925 22:48:12.876113 13091 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:48:12.879820 13098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff36b9e4d624ea5b6b38443ef7477f7. Latest consensus state: current_term: 1 leader_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } } }
I20260925 22:48:12.880108 13098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:12.879516 13096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } } }
I20260925 22:48:12.880923 13096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:12.881677 13113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:12.885538 13113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:12.895843 13113 catalog_manager.cc:1347] Generated new cluster ID: 42911ba139e3446fbfedd1e98065d1f3
I20260925 22:48:12.895910 13113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:12.910574 13113 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:12.911938 13113 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:12.920578 13113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: Generated new TSK 0
I20260925 22:48:12.921454 13113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:48:13.571125 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:57221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33675
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:13.689100 13136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:13.689518 13136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:13.689632 13136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:13.694126 13136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:13.694213 13136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:13.694319 13136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:48:13.700052 13136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.201.148:57221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:13.701205 13136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:13.702247 13136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:13.712733 13141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:13.712707 13144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:13.712847 13136 server_base.cc:1034] running on GCE node
W20260925 22:48:13.712703 13142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:13.713451 13136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:13.714095 13136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:13.715272 13136 hybrid_clock.cc:648] HybridClock initialized: now 1790376493715241 us; error 40 us; skew 500 ppm
Sep 25 22:48:13 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:13.718338 13136 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:48:13.719388 13136 webserver.cc:460] Webserver started at http://127.0.201.129:44203/ using document root <none> and password file <none>
I20260925 22:48:13.720113 13136 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:13.720204 13136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:13.720440 13136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:48:13 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:13.735719 13136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "e2c44d00e0bd43fbb2f61db6b69ff311"
format_stamp: "Formatted at 2026-09-25 22:48:13 on dist-test-slave-vdtf"
server_key: "6ddebba94c9af8263fa019975d5475e0"
server_key_iv: "0a15d596215801945ec190f5328c7629"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:13.736567 13136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "e2c44d00e0bd43fbb2f61db6b69ff311"
format_stamp: "Formatted at 2026-09-25 22:48:13 on dist-test-slave-vdtf"
server_key: "6ddebba94c9af8263fa019975d5475e0"
server_key_iv: "0a15d596215801945ec190f5328c7629"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:13.742616 13136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 22:48:13.762984 13153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:13.764791 13136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:13.764993 13136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "e2c44d00e0bd43fbb2f61db6b69ff311"
format_stamp: "Formatted at 2026-09-25 22:48:13 on dist-test-slave-vdtf"
server_key: "6ddebba94c9af8263fa019975d5475e0"
server_key_iv: "0a15d596215801945ec190f5328c7629"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:13.765208 13136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:13.801350 13136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:13.805195 13136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:13.805452 13136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:13.806169 13136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:13.807173 13136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:13.807276 13136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:13.807399 13136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:13.807494 13136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:13.820284 13136 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:39799
I20260925 22:48:13.821509 13136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:13.830094 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 13136
I20260925 22:48:13.830225 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:48:13.830528 806 external_mini_cluster.cc:1428] Setting key 47f4918366b0d20c158a33bd777e5fca
I20260925 22:48:13.823637 13266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:39799 every 8 connection(s)
Sep 25 22:48:13 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:13.839504 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:13.825792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:34017 (local address 127.0.201.190:33675)
0925 22:48:13.826157 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:13.826165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:13.826181 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:48:13.827386 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:13.827389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:13.827460 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:13.827547 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:13.829080 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:13.829942 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:13.831885 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:13.832219 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:13.832293 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:13.835149 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:13.835152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:13.835156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:13.835180 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:13.837027 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:13.837735 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:13.837739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:13.837741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:13.837828 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:13.838217 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:13.838220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:13.838222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:13.838444 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:13.838516 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:13.839187 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 22:48:13.839312 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":98,"threads_started":1}
I20260925 22:48:13.841084 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:13.824603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33675 (local address 127.0.201.129:34017)
0925 22:48:13.825973 (+ 1370us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:13.826035 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:13.827048 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:13.827884 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:13.827892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:13.828240 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:13.828911 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:13.828930 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:13.830110 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:13.830113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:13.831738 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:13.831752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:13.832494 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:13.832498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:13.832682 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:13.833335 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:13.833356 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:13.834962 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:13.837160 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:13.837166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:13.837183 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:13.837623 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:13.837938 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:13.837940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:13.837942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:13.838109 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:13.838598 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:13.838604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:13.838902 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 22:48:13.839703 (+ 801us) client_negotiation.cc:240] Negotiation successful
0925 22:48:13.840033 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"thread_start_us":128,"threads_started":1}
I20260925 22:48:13.844144 13267 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33675
I20260925 22:48:13.844476 13267 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:13.845400 13267 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
I20260925 22:48:13.847394 13055 ts_manager.cc:194] Registered new tserver with Master: e2c44d00e0bd43fbb2f61db6b69ff311 (127.0.201.129:39799)
I20260925 22:48:13.849022 13055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:34017
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:48:13.898574 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:57221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33675
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:48:13 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 22:48:14.058081 13275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:14.058444 13275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:14.058507 13275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:14.064620 13275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:14.064787 13275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:14.064919 13275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:48:14.072950 13275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.201.148:57221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:14.074714 13275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:14.076336 13275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:14.093478 13284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:14.093755 13281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:14.093468 13282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:14.095891 13275 server_base.cc:1034] running on GCE node
I20260925 22:48:14.096410 13275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:14.097224 13275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:14.104632 13275 hybrid_clock.cc:648] HybridClock initialized: now 1790376494104550 us; error 89 us; skew 500 ppm
Sep 25 22:48:14 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:14.108752 13275 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:48:14.110248 13275 webserver.cc:460] Webserver started at http://127.0.201.130:41249/ using document root <none> and password file <none>
I20260925 22:48:14.111085 13275 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:14.111244 13275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:14.111562 13275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:48:14 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:14.126569 13275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "b3a742219ba44019bbeb99366d13dc7f"
format_stamp: "Formatted at 2026-09-25 22:48:14 on dist-test-slave-vdtf"
server_key: "d50e17d709c7c70a5a966bcaa57742d6"
server_key_iv: "3ac3c30c949c0b5eb2f931e35a652319"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:14.127282 13275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "b3a742219ba44019bbeb99366d13dc7f"
format_stamp: "Formatted at 2026-09-25 22:48:14 on dist-test-slave-vdtf"
server_key: "d50e17d709c7c70a5a966bcaa57742d6"
server_key_iv: "3ac3c30c949c0b5eb2f931e35a652319"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:14.140458 13275 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.000s
I20260925 22:48:14.161298 13291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:14.163030 13275 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260925 22:48:14.163182 13275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "b3a742219ba44019bbeb99366d13dc7f"
format_stamp: "Formatted at 2026-09-25 22:48:14 on dist-test-slave-vdtf"
server_key: "d50e17d709c7c70a5a966bcaa57742d6"
server_key_iv: "3ac3c30c949c0b5eb2f931e35a652319"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:14.163334 13275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:14.199476 13275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:14.202823 13275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:14.203028 13275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:14.203753 13275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:14.204802 13275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:14.204869 13275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:14.204932 13275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:14.204964 13275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:14.215036 13275 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:44773
I20260925 22:48:14.216279 13275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:14.220973 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 13275
I20260925 22:48:14.221129 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:48:14.221504 806 external_mini_cluster.cc:1428] Setting key ff243dfd23eded2070bc41e08f5d68fc
I20260925 22:48:14.224815 13404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:44773 every 8 connection(s)
Sep 25 22:48:14 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:48:14.244362 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:14.229409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:55655 (local address 127.0.201.190:33675)
0925 22:48:14.229548 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:14.229552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:14.229994 (+ 442us) server_negotiation.cc:407] Connection header received
0925 22:48:14.231347 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:14.231352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:14.231415 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:14.231485 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:14.233333 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:14.234174 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:14.235958 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:14.236330 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:14.236431 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:14.239618 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:14.239622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:14.239625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:14.239656 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:14.241655 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:14.242303 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:14.242307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:14.242309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:14.242399 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:14.242855 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:14.242858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:14.242860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:14.243101 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:14.243193 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:14.243969 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 22:48:14.244154 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:48:14.246093 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:14.228304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33675 (local address 127.0.201.130:55655)
0925 22:48:14.229832 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:14.229891 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:14.231093 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:14.231974 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:14.231984 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:14.232435 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:14.233140 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:14.233171 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:14.234337 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:14.234341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:14.235805 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:14.235818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:14.236651 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:14.236655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:14.236868 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:14.237573 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:14.237596 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:14.239384 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:14.241805 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:14.241811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:14.241829 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:14.242173 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:14.242535 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:14.242538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:14.242541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:14.242737 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:14.243265 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:14.243272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:14.243636 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 22:48:14.244623 (+ 987us) client_negotiation.cc:240] Negotiation successful
0925 22:48:14.244992 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"thread_start_us":160,"threads_started":1}
I20260925 22:48:14.247454 13405 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33675
I20260925 22:48:14.247727 13405 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:14.248323 13405 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
I20260925 22:48:14.249414 13055 ts_manager.cc:194] Registered new tserver with Master: b3a742219ba44019bbeb99366d13dc7f (127.0.201.130:44773)
I20260925 22:48:14.250186 13055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:55655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:48:14.278343 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:57221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33675
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:14.437979 13410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:14.438395 13410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:14.438505 13410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:14.444288 13410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:14.444474 13410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:14.444607 13410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:48:14.452421 13410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.201.148:57221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:14.453925 13410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:14.455305 13410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:14.468061 13416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:14.468142 13410 server_base.cc:1034] running on GCE node
W20260925 22:48:14.468523 13415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:14.468852 13418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:14.469388 13410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:14.470157 13410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:14.471305 13410 hybrid_clock.cc:648] HybridClock initialized: now 1790376494471212 us; error 107 us; skew 500 ppm
Sep 25 22:48:14 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:14.474924 13410 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:48:14.476065 13410 webserver.cc:460] Webserver started at http://127.0.201.131:33063/ using document root <none> and password file <none>
I20260925 22:48:14.476830 13410 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:14.476969 13410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:14.477252 13410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:48:14 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:14.494861 13410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "4ab945be52ec4586ab4a41287db05579"
format_stamp: "Formatted at 2026-09-25 22:48:14 on dist-test-slave-vdtf"
server_key: "15b1270696272a8a9712027334b0ed1d"
server_key_iv: "79089c4767d55b82afe622af9e65aaba"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:14.495872 13410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "4ab945be52ec4586ab4a41287db05579"
format_stamp: "Formatted at 2026-09-25 22:48:14 on dist-test-slave-vdtf"
server_key: "15b1270696272a8a9712027334b0ed1d"
server_key_iv: "79089c4767d55b82afe622af9e65aaba"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:14.525486 13410 fs_manager.cc:691] Time spent creating directory manager: real 0.029s user 0.003s sys 0.000s
Sep 25 22:48:14 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:14.545903 13425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:14.547598 13410 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260925 22:48:14.547746 13410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "4ab945be52ec4586ab4a41287db05579"
format_stamp: "Formatted at 2026-09-25 22:48:14 on dist-test-slave-vdtf"
server_key: "15b1270696272a8a9712027334b0ed1d"
server_key_iv: "79089c4767d55b82afe622af9e65aaba"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:14.547895 13410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:14.570132 13410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:14.574261 13410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:14.574505 13410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:14.575444 13410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:14.576617 13410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:14.576678 13410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:14.576820 13410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:14.576849 13410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:14.589917 13410 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:36797
I20260925 22:48:14.591115 13410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:14.591511 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 13410
I20260925 22:48:14.591616 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:48:14.591954 806 external_mini_cluster.cc:1428] Setting key 3f9b0d2cbc0d00a0bd3828591e9ac737
I20260925 22:48:14.594561 13538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:36797 every 8 connection(s)
Sep 25 22:48:14 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:14.628113 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:14.598568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:37279 (local address 127.0.201.190:33675)
0925 22:48:14.598686 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:14.598691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:14.599171 (+ 480us) server_negotiation.cc:407] Connection header received
0925 22:48:14.600461 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:14.600465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:14.600518 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:14.600569 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:14.602291 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:14.603160 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:14.618367 (+ 15207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:14.618749 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:14.618832 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:14.622577 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:14.622582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:14.622584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:14.622615 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:14.624439 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:14.625129 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:14.625133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:14.625136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:14.625222 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:14.625640 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:14.625643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:14.625645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:14.625897 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:14.626007 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:14.627812 (+ 1805us) server_negotiation.cc:299] Negotiation successful
0925 22:48:14.627955 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:48:14.629609 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:14.597524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33675 (local address 127.0.201.131:37279)
0925 22:48:14.599026 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:14.599082 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:14.600219 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:14.600992 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:14.601001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:14.601360 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:14.602112 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:14.602132 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:14.603320 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:14.603323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:14.618204 (+ 14881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:14.618222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:14.619070 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:14.619074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:14.619310 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:14.620089 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:14.620114 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:14.622350 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:14.624586 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:14.624595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:14.624618 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:14.625007 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:14.625318 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:14.625321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:14.625323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:14.625524 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:14.626063 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:14.626073 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:14.626420 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 22:48:14.626627 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 22:48:14.627019 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1288,"thread_start_us":155,"threads_started":1}
I20260925 22:48:14.631246 13539 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33675
I20260925 22:48:14.631486 13539 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:14.632184 13539 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
I20260925 22:48:14.633407 13055 ts_manager.cc:194] Registered new tserver with Master: 4ab945be52ec4586ab4a41287db05579 (127.0.201.131:36797)
I20260925 22:48:14.634174 13055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:37279
I20260925 22:48:14.640736 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:14.641645 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 13136
I20260925 22:48:14.649346 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 13275
I20260925 22:48:14.657348 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 13410
I20260925 22:48:14.665486 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 13023
I20260925 22:48:14.673592 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:33675
--webserver_interface=127.0.201.190
--webserver_port=44743
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:33675
--ranger_config_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:57221/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.201.148:57221/kms with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:14.793402 13542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:14.793658 13542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:14.793711 13542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:14.797598 13542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:14.797663 13542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:14.797683 13542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:14.797698 13542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:14.797712 13542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:14.802965 13542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.201.148:57221/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:33675
--ranger_config_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:33675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=44743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:14.803869 13542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:14.804843 13542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:14.812968 13550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:14.812996 13548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:14.812999 13547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:14.813230 13542 server_base.cc:1034] running on GCE node
I20260925 22:48:14.813853 13542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:14.814985 13542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:14.816151 13542 hybrid_clock.cc:648] HybridClock initialized: now 1790376494816131 us; error 32 us; skew 500 ppm
Sep 25 22:48:14 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:14.818856 13542 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:48:14.819749 13542 webserver.cc:460] Webserver started at http://127.0.201.190:44743/ using document root <none> and password file <none>
I20260925 22:48:14.820273 13542 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:14.820376 13542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:48:14 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:14.891532 13542 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:48:14.894398 13558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:14.895957 13542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:14.896087 13542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "8ff36b9e4d624ea5b6b38443ef7477f7"
format_stamp: "Formatted at 2026-09-25 22:48:12 on dist-test-slave-vdtf"
server_key: "b033e7416e4d0048609a4a5a75705c41"
server_key_iv: "aef3e2f1ac9ec78b74bb33f7ba818673"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:14.896798 13542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:14.921849 13542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:14.925297 13542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:14.925540 13542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:14.934414 13542 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:33675
I20260925 22:48:14.934446 13610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:33675 every 8 connection(s)
I20260925 22:48:14.935757 13542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:14.939977 13611 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:48:14.941661 13611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: Bootstrap starting.
I20260925 22:48:14.942632 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 13542
I20260925 22:48:14.943148 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:57221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:39799
--local_ip_for_outbound_sockets=127.0.201.129
--tserver_master_addrs=127.0.201.190:33675
--webserver_port=44203
--webserver_interface=127.0.201.129
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.201.148:57221/kms with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:48:14.957914 13611 log.cc:824] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:14.964349 13611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:48:14.964732 13611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: Bootstrap complete.
I20260925 22:48:14.968921 13611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } }
I20260925 22:48:14.969390 13611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff36b9e4d624ea5b6b38443ef7477f7, State: Initialized, Role: FOLLOWER
I20260925 22:48:14.969879 13611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } }
I20260925 22:48:14.970023 13611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:14.970111 13611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:14.970227 13611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:48:14.972204 13611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } }
I20260925 22:48:14.972539 13611 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff36b9e4d624ea5b6b38443ef7477f7; no voters:
I20260925 22:48:14.972873 13611 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:48:14.972990 13615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:48:14.973196 13615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [term 2 LEADER]: Becoming Leader. State: Replica: 8ff36b9e4d624ea5b6b38443ef7477f7, State: Running, Role: LEADER
I20260925 22:48:14.973484 13615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } }
I20260925 22:48:14.974097 13611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:14.974756 13616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } } }
I20260925 22:48:14.974879 13616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:14.975191 13617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff36b9e4d624ea5b6b38443ef7477f7. Latest consensus state: current_term: 2 leader_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff36b9e4d624ea5b6b38443ef7477f7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33675 } } }
I20260925 22:48:14.975289 13617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:14.975950 13621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:14.976477 13611 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:48:14.982066 13621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:14.982872 13621 catalog_manager.cc:1259] Loaded cluster ID: 42911ba139e3446fbfedd1e98065d1f3
I20260925 22:48:14.982945 13621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:14.985543 13621 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:14.986220 13621 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: Loaded TSK: 0
I20260925 22:48:14.987425 13621 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:48:15.071064 13613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:15.071336 13613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:15.071412 13613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:15.075749 13613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:15.075825 13613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:15.075932 13613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:48:15.083611 13613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.201.148:57221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:39799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=44203
--tserver_master_addrs=127.0.201.190:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:15.085026 13613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:15.086158 13613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:15.096309 13649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:15.096321 13652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:15.096958 13613 server_base.cc:1034] running on GCE node
W20260925 22:48:15.099841 13650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:15.100282 13613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:15.100970 13613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:15.102211 13613 hybrid_clock.cc:648] HybridClock initialized: now 1790376495102173 us; error 44 us; skew 500 ppm
Sep 25 22:48:15 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:15.105540 13613 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:48:15.106653 13613 webserver.cc:460] Webserver started at http://127.0.201.129:44203/ using document root <none> and password file <none>
I20260925 22:48:15.107334 13613 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:15.107446 13613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:48:15 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:15.141508 13613 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260925 22:48:15.145402 13660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:15.146806 13613 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:48:15.146961 13613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "e2c44d00e0bd43fbb2f61db6b69ff311"
format_stamp: "Formatted at 2026-09-25 22:48:13 on dist-test-slave-vdtf"
server_key: "6ddebba94c9af8263fa019975d5475e0"
server_key_iv: "0a15d596215801945ec190f5328c7629"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:15.147549 13613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:15.181409 13613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:15.185205 13613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:15.185426 13613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:15.186131 13613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:15.187171 13613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:15.187238 13613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:15.187304 13613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:15.187345 13613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:15.198151 13613 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:39799
I20260925 22:48:15.199397 13613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:15.205739 13773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:39799 every 8 connection(s)
I20260925 22:48:15.206686 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 13613
I20260925 22:48:15.207192 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:57221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:44773
--local_ip_for_outbound_sockets=127.0.201.130
--tserver_master_addrs=127.0.201.190:33675
--webserver_port=41249
--webserver_interface=127.0.201.130
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.201.148:57221/kms with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:48:15 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:15.228997 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:15.209385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33675 (local address 127.0.201.129:48147)
0925 22:48:15.210548 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:15.210605 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:15.211671 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:15.213357 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:15.213366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:15.213711 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:15.214328 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:15.214347 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.216256 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.216260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:15.217692 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:15.217702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.218920 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.218924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:15.219105 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:15.219824 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:15.219844 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:15.222416 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:15.225459 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:15.225465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:15.225480 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:15.225810 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:15.226204 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:15.226207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:15.226209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:15.226396 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:15.227143 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:15.227149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:15.227419 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 22:48:15.227625 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 22:48:15.227975 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":138,"threads_started":1}
I20260925 22:48:15.229224 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:15.208281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:48147 (local address 127.0.201.190:33675)
0925 22:48:15.210711 (+ 2430us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:15.210734 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:15.210783 (+ 49us) server_negotiation.cc:407] Connection header received
0925 22:48:15.212168 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:15.212183 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:15.212533 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:15.212885 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:15.214523 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.216095 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.217838 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.218548 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.218703 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:15.222639 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:15.222646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:15.222663 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:15.222696 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:15.225323 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:15.225943 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:15.225949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:15.225955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:15.226094 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:15.226508 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:15.226511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:15.226513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:15.226967 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:15.227091 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:15.227824 (+ 733us) server_negotiation.cc:299] Negotiation successful
0925 22:48:15.228126 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2243,"thread_start_us":111,"threads_started":1}
I20260925 22:48:15.232333 13774 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33675
I20260925 22:48:15.232551 13774 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:15.233084 13774 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
W20260925 22:48:15.235260 13774 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:33675 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:48:15.235366 13774 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:15.235519 13774 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
I20260925 22:48:15.236406 13774 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:15.236554 13774 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
W20260925 22:48:15.237277 13774 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:48:15.350108 13776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:15.350436 13776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:15.350504 13776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:15.356590 13776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:15.356663 13776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:15.356748 13776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:48:15.362032 13776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.201.148:57221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:44773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=41249
--tserver_master_addrs=127.0.201.190:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:15.363049 13776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:15.364111 13776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:15.376080 13783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:15.375977 13786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:15.376147 13776 server_base.cc:1034] running on GCE node
W20260925 22:48:15.375990 13784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:15.376643 13776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:15.377267 13776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:15.378466 13776 hybrid_clock.cc:648] HybridClock initialized: now 1790376495378437 us; error 46 us; skew 500 ppm
Sep 25 22:48:15 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:15.386238 13776 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:48:15.387377 13776 webserver.cc:460] Webserver started at http://127.0.201.130:41249/ using document root <none> and password file <none>
I20260925 22:48:15.388195 13776 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:15.388296 13776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:48:15 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:15.424153 13776 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.001s
I20260925 22:48:15.432354 13794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:15.436964 13776 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260925 22:48:15.437124 13776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "b3a742219ba44019bbeb99366d13dc7f"
format_stamp: "Formatted at 2026-09-25 22:48:14 on dist-test-slave-vdtf"
server_key: "d50e17d709c7c70a5a966bcaa57742d6"
server_key_iv: "3ac3c30c949c0b5eb2f931e35a652319"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:15.437927 13776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:15.464787 13776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:15.468519 13776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:15.468716 13776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:15.469441 13776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:15.470504 13776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:15.470564 13776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:15.470618 13776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:15.470642 13776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:15.482426 13776 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:44773
I20260925 22:48:15.482600 13908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:44773 every 8 connection(s)
I20260925 22:48:15.483718 13776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:15.490787 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 13776
I20260925 22:48:15.491297 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.201.148:57221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:36797
--local_ip_for_outbound_sockets=127.0.201.131
--tserver_master_addrs=127.0.201.190:33675
--webserver_port=33063
--webserver_interface=127.0.201.131
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.201.148:57221/kms with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:48:15 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:15.498155 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:15.485922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:57303 (local address 127.0.201.190:33675)
0925 22:48:15.486073 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:15.486078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:15.486879 (+ 801us) server_negotiation.cc:407] Connection header received
0925 22:48:15.487857 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:15.487860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:15.487919 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:15.487994 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:15.489083 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.489613 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.490903 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.491275 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.491410 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:15.494090 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:15.494093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:15.494096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:15.494128 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:15.495569 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:15.496223 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:15.496227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:15.496229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:15.496313 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:15.496708 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:15.496712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:15.496714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:15.496948 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:15.497035 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:15.497762 (+ 727us) server_negotiation.cc:299] Negotiation successful
0925 22:48:15.497922 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:48:15.498957 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:15.486214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33675 (local address 127.0.201.130:57303)
0925 22:48:15.486710 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:15.486762 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:15.487585 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:15.488165 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:15.488172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:15.488473 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:15.488936 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:15.488949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.489738 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.489743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:15.490750 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:15.490763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.491440 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.491447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:15.491717 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:15.492454 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:15.492479 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:15.493913 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:15.495761 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:15.495766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:15.495780 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:15.496103 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:15.496424 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:15.496427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:15.496429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:15.496595 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:15.497104 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:15.497111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:15.497415 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 22:48:15.497646 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 22:48:15.497978 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":151,"threads_started":1}
I20260925 22:48:15.500358 13909 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33675
I20260925 22:48:15.500602 13909 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:15.501127 13909 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
W20260925 22:48:15.502774 13909 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:33675 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:48:15.502902 13909 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:15.503073 13909 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
I20260925 22:48:15.503835 13909 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:15.504032 13909 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
W20260925 22:48:15.504953 13909 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:48:15.655750 13912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:15.656145 13912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:15.656222 13912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:15.662849 13912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:15.662932 13912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:15.663034 13912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:48:15.671279 13912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.201.148:57221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:36797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=33063
--tserver_master_addrs=127.0.201.190:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:15.672736 13912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:15.674120 13912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:15.686630 13920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:15.686791 13912 server_base.cc:1034] running on GCE node
W20260925 22:48:15.686620 13917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:15.686621 13918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:15.687466 13912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:15.688256 13912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:15.691581 13912 hybrid_clock.cc:648] HybridClock initialized: now 1790376495691541 us; error 46 us; skew 500 ppm
Sep 25 22:48:15 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:15.694990 13912 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:48:15.695998 13912 webserver.cc:460] Webserver started at http://127.0.201.131:33063/ using document root <none> and password file <none>
I20260925 22:48:15.696669 13912 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:15.696785 13912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:48:15 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:15.717164 13912 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260925 22:48:15.721130 13928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:15.722658 13912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:48:15.722796 13912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "4ab945be52ec4586ab4a41287db05579"
format_stamp: "Formatted at 2026-09-25 22:48:14 on dist-test-slave-vdtf"
server_key: "15b1270696272a8a9712027334b0ed1d"
server_key_iv: "79089c4767d55b82afe622af9e65aaba"
server_key_version: "kuduclusterkey@0"
I20260925 22:48:15.723500 13912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:15.757745 13912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:15.761366 13912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:15.761562 13912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:15.762239 13912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:15.763263 13912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:15.763331 13912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:15.763394 13912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:15.763432 13912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:15.774575 13912 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:36797
I20260925 22:48:15.774744 14048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:36797 every 8 connection(s)
I20260925 22:48:15.775746 13912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:15.780300 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 13912
Sep 25 22:48:15 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:15.791471 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:15.781015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.0.201.190:33675)
0925 22:48:15.781280 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:15.781288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:15.781549 (+ 261us) server_negotiation.cc:407] Connection header received
0925 22:48:15.781657 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:15.781660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:15.781708 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:15.781802 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:15.782659 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.783152 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.784389 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.784857 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.784998 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:15.786998 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:15.787005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:15.787010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:15.787041 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:15.788562 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:15.789046 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:15.789051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:15.789054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:15.789140 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:15.789633 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:15.789638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:15.789641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:15.789941 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:15.790204 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:15.790359 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 22:48:15.790536 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":192,"spinlock_wait_cycles":2816,"thread_start_us":110,"threads_started":1}
I20260925 22:48:15.793296 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:15.778180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:37831 (local address 127.0.201.190:33675)
0925 22:48:15.778317 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:15.778321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:15.779277 (+ 956us) server_negotiation.cc:407] Connection header received
0925 22:48:15.780388 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:15.780392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:15.780449 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:15.780572 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:15.782007 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.782535 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.784344 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.784829 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.784996 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:15.788810 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:15.788814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:15.788817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:15.788843 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:15.790742 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:15.791386 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:15.791391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:15.791393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:15.791482 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:15.791980 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:15.791984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:15.791986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:15.792223 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:15.792323 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:15.792949 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 22:48:15.793066 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":5248}
I20260925 22:48:15.794085 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:15.778615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33675 (local address 127.0.201.131:37831)
0925 22:48:15.779095 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:15.779147 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:15.780140 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:15.780810 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:15.780822 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:15.781202 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:15.781864 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:15.781886 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.783272 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.783275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:15.784223 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:15.784233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:15.785001 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:15.785005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:15.785201 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:15.785885 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:15.785911 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:15.788047 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:15.790880 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:15.790886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:15.790901 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:15.791252 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:15.791613 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:15.791617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:15.791619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:15.791875 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:15.792383 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:15.792390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:15.792676 (+ 286us) client_negotiation.cc:769] Sending connection context
0925 22:48:15.792889 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 22:48:15.793234 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":145,"threads_started":1}
I20260925 22:48:15.795218 14049 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33675
I20260925 22:48:15.795466 14049 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:15.796057 14049 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
I20260925 22:48:15.797470 13575 ts_manager.cc:194] Registered new tserver with Master: 4ab945be52ec4586ab4a41287db05579 (127.0.201.131:36797)
I20260925 22:48:15.799073 13575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:37831
Sep 25 22:48:16 dist-test-slave-vdtf krb5kdc[12579](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:16.237514 13774 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:16.237849 13774 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
I20260925 22:48:16.238656 13574 ts_manager.cc:194] Registered new tserver with Master: e2c44d00e0bd43fbb2f61db6b69ff311 (127.0.201.129:39799)
I20260925 22:48:16.239493 13574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:48147
I20260925 22:48:16.505172 13909 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:16.505446 13909 heartbeater.cc:511] Master 127.0.201.190:33675 requested a full tablet report, sending...
I20260925 22:48:16.511963 13574 ts_manager.cc:194] Registered new tserver with Master: b3a742219ba44019bbeb99366d13dc7f (127.0.201.130:44773)
I20260925 22:48:16.512737 13574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:57303
I20260925 22:48:16.515664 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:16.526851 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.517156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40986 (local address 127.0.201.190:33675)
0925 22:48:16.517443 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:16.517448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:16.517787 (+ 339us) server_negotiation.cc:407] Connection header received
0925 22:48:16.517909 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:16.517912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:16.517961 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:16.518049 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:16.518910 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.519666 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.520859 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.521219 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.521336 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.522907 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:16.522911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:16.522914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:16.522941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:16.524673 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:16.525246 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:16.525250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:16.525252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:16.525342 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:16.525627 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:16.525630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:16.525632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:16.525833 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:16.525953 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:16.526566 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 22:48:16.526687 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":111,"threads_started":1}
I20260925 22:48:16.531002 13574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:48:16.535641 13574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 25 22:48:16 dist-test-slave-vdtf krb5kdc[12579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for HTTP/127.0.201.148@KRBTEST.COM
I20260925 22:48:16.554242 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.546946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:44773 (local address 127.0.0.1:36622)
0925 22:48:16.548480 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:16.548529 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:16.548643 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:16.549523 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:16.549529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:16.549552 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:16.549698 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:16.549704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.551212 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.551216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:16.552548 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:16.552559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.553684 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.553688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:16.553815 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.553863 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 22:48:16.553955 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:48:16.554075 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"thread_start_us":90,"threads_started":1}
I20260925 22:48:16.555240 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.548343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.0.201.130:44773)
0925 22:48:16.549057 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:16.549063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:16.549084 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:48:16.549153 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:16.549158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:16.549365 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:16.549843 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:16.549973 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.551068 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.552712 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.553542 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.554788 (+ 1246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.554942 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:16.554995 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 22:48:16.555087 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":95,"threads_started":1}
I20260925 22:48:16.558596 13842 tablet_service.cc:1467] Processing CreateTablet for tablet 087ea0527d614171b1ff3cfce4a3f776 (DEFAULT_TABLE table=test-table [id=e73f871d3e51456abba5a76494ce5499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:16.559654 13842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087ea0527d614171b1ff3cfce4a3f776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:16.561352 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.547976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36797 (local address 127.0.0.1:44924)
0925 22:48:16.555877 (+ 7901us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:16.555892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:16.556113 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:16.556806 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:16.556809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:16.556826 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:16.556918 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:16.556923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.558317 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.558320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:16.559520 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:16.559532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.560917 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.560921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:16.561011 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.561034 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:48:16.561110 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:48:16.561180 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7838,"thread_start_us":72,"threads_started":1}
I20260925 22:48:16.561905 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.548149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.0.201.131:36797)
0925 22:48:16.548896 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:16.548902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:16.555985 (+ 7083us) server_negotiation.cc:407] Connection header received
0925 22:48:16.556283 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:16.556290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:16.556505 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:16.556628 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:16.557041 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.558157 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.559669 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.560589 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.560703 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.560838 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:16.561650 (+ 812us) server_negotiation.cc:299] Negotiation successful
0925 22:48:16.561764 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":122,"threads_started":1}
I20260925 22:48:16.564883 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.547660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:39799 (local address 127.0.0.1:51446)
0925 22:48:16.554404 (+ 6744us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:16.554418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:16.554491 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:16.560304 (+ 5813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:16.560307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:16.560326 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:16.560407 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:16.560413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.561993 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.561996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:16.563106 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:16.563117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.564527 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.564531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:16.564622 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.564648 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:48:16.564694 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 22:48:16.564759 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6687,"thread_start_us":84,"threads_started":1}
I20260925 22:48:16.565565 14071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.547891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.0.201.129:39799)
0925 22:48:16.559842 (+ 11951us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:16.559848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:16.559872 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:48:16.559945 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:16.559951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:16.560140 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:16.560560 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:16.560706 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.561863 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.563443 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.564390 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.565133 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.565290 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:16.565341 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 22:48:16.565431 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11834,"thread_start_us":99,"threads_started":1}
I20260925 22:48:16.566264 13976 tablet_service.cc:1467] Processing CreateTablet for tablet 087ea0527d614171b1ff3cfce4a3f776 (DEFAULT_TABLE table=test-table [id=e73f871d3e51456abba5a76494ce5499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:16.567373 13976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087ea0527d614171b1ff3cfce4a3f776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:16.568357 13703 tablet_service.cc:1467] Processing CreateTablet for tablet 087ea0527d614171b1ff3cfce4a3f776 (DEFAULT_TABLE table=test-table [id=e73f871d3e51456abba5a76494ce5499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:16.569396 13703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087ea0527d614171b1ff3cfce4a3f776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:16.578184 14076 tablet_bootstrap.cc:492] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311: Bootstrap starting.
I20260925 22:48:16.580484 14076 tablet_bootstrap.cc:654] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:16.581346 14076 log.cc:824] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:16.583194 14076 tablet_bootstrap.cc:492] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311: No bootstrap required, opened a new log
I20260925 22:48:16.583421 14076 ts_tablet_manager.cc:1397] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:48:16.584954 14075 tablet_bootstrap.cc:492] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: Bootstrap starting.
I20260925 22:48:16.587260 14075 tablet_bootstrap.cc:654] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:16.587726 14076 raft_consensus.cc:348] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 } } peers { permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 } } peers { permanent_uuid: "4ab945be52ec4586ab4a41287db05579" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36797 } }
I20260925 22:48:16.588007 14076 raft_consensus.cc:374] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:16.588063 14076 raft_consensus.cc:729] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c44d00e0bd43fbb2f61db6b69ff311, State: Initialized, Role: FOLLOWER
I20260925 22:48:16.588186 14075 log.cc:824] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:16.588569 14076 consensus_queue.cc:260] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 } } peers { permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 } } peers { permanent_uuid: "4ab945be52ec4586ab4a41287db05579" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36797 } }
I20260925 22:48:16.588963 14078 tablet_bootstrap.cc:492] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: Bootstrap starting.
I20260925 22:48:16.589646 14076 ts_tablet_manager.cc:1428] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:48:16.590018 13774 heartbeater.cc:503] Master 127.0.201.190:33675 was elected leader, sending a full tablet report...
I20260925 22:48:16.591218 14078 tablet_bootstrap.cc:654] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:16.591974 14078 log.cc:824] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:16.594017 14075 tablet_bootstrap.cc:492] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: No bootstrap required, opened a new log
I20260925 22:48:16.594187 14075 ts_tablet_manager.cc:1397] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260925 22:48:16.598383 14075 raft_consensus.cc:348] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 } } peers { permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 } } peers { permanent_uuid: "4ab945be52ec4586ab4a41287db05579" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36797 } }
I20260925 22:48:16.598644 14075 raft_consensus.cc:374] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:16.598685 14075 raft_consensus.cc:729] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab945be52ec4586ab4a41287db05579, State: Initialized, Role: FOLLOWER
I20260925 22:48:16.599149 14075 consensus_queue.cc:260] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 } } peers { permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 } } peers { permanent_uuid: "4ab945be52ec4586ab4a41287db05579" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36797 } }
I20260925 22:48:16.600175 14075 ts_tablet_manager.cc:1428] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:48:16.600543 14049 heartbeater.cc:503] Master 127.0.201.190:33675 was elected leader, sending a full tablet report...
I20260925 22:48:16.602531 14078 tablet_bootstrap.cc:492] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: No bootstrap required, opened a new log
I20260925 22:48:16.602733 14078 ts_tablet_manager.cc:1397] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260925 22:48:16.606096 14078 raft_consensus.cc:348] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 } } peers { permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 } } peers { permanent_uuid: "4ab945be52ec4586ab4a41287db05579" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36797 } }
I20260925 22:48:16.606352 14078 raft_consensus.cc:374] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:16.606416 14078 raft_consensus.cc:729] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a742219ba44019bbeb99366d13dc7f, State: Initialized, Role: FOLLOWER
I20260925 22:48:16.606907 14078 consensus_queue.cc:260] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 } } peers { permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 } } peers { permanent_uuid: "4ab945be52ec4586ab4a41287db05579" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36797 } }
I20260925 22:48:16.607831 13909 heartbeater.cc:503] Master 127.0.201.190:33675 was elected leader, sending a full tablet report...
I20260925 22:48:16.607995 14078 ts_tablet_manager.cc:1428] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 22:48:16.683486 14082 raft_consensus.cc:482] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:16.683750 14082 raft_consensus.cc:504] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 } } peers { permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 } } peers { permanent_uuid: "4ab945be52ec4586ab4a41287db05579" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36797 } }
I20260925 22:48:16.685225 14082 leader_election.cc:288] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3a742219ba44019bbeb99366d13dc7f (127.0.201.130:44773), e2c44d00e0bd43fbb2f61db6b69ff311 (127.0.201.129:39799)
I20260925 22:48:16.691910 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.686500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:60037 (local address 127.0.201.130:44773)
0925 22:48:16.686797 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:16.686802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:16.687622 (+ 820us) server_negotiation.cc:407] Connection header received
0925 22:48:16.687816 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:16.687819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:16.687870 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:16.687922 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:16.688363 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.689103 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.690623 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.691174 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.691272 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.691325 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:16.691700 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 22:48:16.691781 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260925 22:48:16.692312 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.685628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:44773 (local address 127.0.201.131:60037)
0925 22:48:16.687480 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:16.687497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:16.687623 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:16.688069 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:16.688073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:16.688091 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:16.688229 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:16.688235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.689282 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.689285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:16.690477 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:16.690488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.691434 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.691437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:16.691531 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.691553 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:48:16.692080 (+ 527us) client_negotiation.cc:240] Negotiation successful
0925 22:48:16.692159 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1781,"thread_start_us":131,"threads_started":1}
I20260925 22:48:16.692440 14071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.686649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:41427 (local address 127.0.201.129:39799)
0925 22:48:16.686893 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:16.686898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:16.687923 (+ 1025us) server_negotiation.cc:407] Connection header received
0925 22:48:16.688116 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:16.688120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:16.688181 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:16.688230 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:16.688676 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.689480 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.691004 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.691604 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.691741 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.691790 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:16.692229 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 22:48:16.692308 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260925 22:48:16.692981 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.686029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:39799 (local address 127.0.201.131:41427)
0925 22:48:16.687831 (+ 1802us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:16.687845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:16.687995 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:16.688365 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:16.688368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:16.688384 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:16.688518 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:16.688536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.689626 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.689629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:16.690863 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:16.690874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.691885 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.691888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:16.691988 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.692102 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 22:48:16.692762 (+ 660us) client_negotiation.cc:240] Negotiation successful
0925 22:48:16.692836 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1721,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260925 22:48:16.693173 13860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087ea0527d614171b1ff3cfce4a3f776" candidate_uuid: "4ab945be52ec4586ab4a41287db05579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a742219ba44019bbeb99366d13dc7f" is_pre_election: true
I20260925 22:48:16.693490 13860 raft_consensus.cc:2454] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ab945be52ec4586ab4a41287db05579 in term 0.
I20260925 22:48:16.693979 13932 leader_election.cc:302] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab945be52ec4586ab4a41287db05579, b3a742219ba44019bbeb99366d13dc7f; no voters:
I20260925 22:48:16.694021 13719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087ea0527d614171b1ff3cfce4a3f776" candidate_uuid: "4ab945be52ec4586ab4a41287db05579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" is_pre_election: true
I20260925 22:48:16.694278 13719 raft_consensus.cc:2454] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ab945be52ec4586ab4a41287db05579 in term 0.
I20260925 22:48:16.694278 14082 raft_consensus.cc:2781] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:16.694373 14082 raft_consensus.cc:482] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:16.694417 14082 raft_consensus.cc:3037] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:16.695832 14082 raft_consensus.cc:504] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 } } peers { permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 } } peers { permanent_uuid: "4ab945be52ec4586ab4a41287db05579" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36797 } }
I20260925 22:48:16.696223 14082 leader_election.cc:288] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [CANDIDATE]: Term 1 election: Requested vote from peers b3a742219ba44019bbeb99366d13dc7f (127.0.201.130:44773), e2c44d00e0bd43fbb2f61db6b69ff311 (127.0.201.129:39799)
I20260925 22:48:16.696751 13860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087ea0527d614171b1ff3cfce4a3f776" candidate_uuid: "4ab945be52ec4586ab4a41287db05579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a742219ba44019bbeb99366d13dc7f"
I20260925 22:48:16.696753 13719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087ea0527d614171b1ff3cfce4a3f776" candidate_uuid: "4ab945be52ec4586ab4a41287db05579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311"
I20260925 22:48:16.696870 13860 raft_consensus.cc:3037] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:16.696870 13719 raft_consensus.cc:3037] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:16.698140 13719 raft_consensus.cc:2454] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab945be52ec4586ab4a41287db05579 in term 1.
I20260925 22:48:16.698143 13860 raft_consensus.cc:2454] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab945be52ec4586ab4a41287db05579 in term 1.
I20260925 22:48:16.698553 13932 leader_election.cc:302] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab945be52ec4586ab4a41287db05579, b3a742219ba44019bbeb99366d13dc7f; no voters:
I20260925 22:48:16.698810 14082 raft_consensus.cc:2781] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:16.699030 14082 raft_consensus.cc:686] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 1 LEADER]: Becoming Leader. State: Replica: 4ab945be52ec4586ab4a41287db05579, State: Running, Role: LEADER
I20260925 22:48:16.699393 14082 consensus_queue.cc:237] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 } } peers { permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 } } peers { permanent_uuid: "4ab945be52ec4586ab4a41287db05579" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36797 } }
I20260925 22:48:16.703065 13574 catalog_manager.cc:5495] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ab945be52ec4586ab4a41287db05579 (127.0.201.131). New cstate: current_term: 1 leader_uuid: "4ab945be52ec4586ab4a41287db05579" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab945be52ec4586ab4a41287db05579" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36797 } health_report { overall_health: HEALTHY } } }
W20260925 22:48:16.710723 13775 tablet.cc:2369] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:16.726166 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.721388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.0.201.131:36797)
0925 22:48:16.721524 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:16.721529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:16.721546 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:48:16.721629 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:16.721633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:16.721698 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:16.721821 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:16.722102 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.722872 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.724092 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:16.724417 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:16.724503 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:16.724883 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:16.725461 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:16.725584 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:16.725963 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 22:48:16.726025 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:48:16.733214 13719 raft_consensus.cc:1261] T 087ea0527d614171b1ff3cfce4a3f776 P e2c44d00e0bd43fbb2f61db6b69ff311 [term 1 FOLLOWER]: Refusing update from remote peer 4ab945be52ec4586ab4a41287db05579: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:16.733578 13860 raft_consensus.cc:1261] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f [term 1 FOLLOWER]: Refusing update from remote peer 4ab945be52ec4586ab4a41287db05579: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:16.734239 14086 consensus_queue.cc:1035] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c44d00e0bd43fbb2f61db6b69ff311" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:16.734669 14086 consensus_queue.cc:1035] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3a742219ba44019bbeb99366d13dc7f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:48:16.738776 13910 tablet.cc:2369] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:16.746691 14089 mvcc.cc:205] Tried to move back new op lower bound from 7333382130611576832 to 7333382130483974144. Current Snapshot: MvccSnapshot[applied={T|T < 7333382130611576832}]
I20260925 22:48:16.752485 14091 mvcc.cc:205] Tried to move back new op lower bound from 7333382130611576832 to 7333382130483974144. Current Snapshot: MvccSnapshot[applied={T|T < 7333382130611576832}]
I20260925 22:48:16.762286 13575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:16.762491 13575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:16.765203 13575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ff36b9e4d624ea5b6b38443ef7477f7: Sending DeleteTablet for 3 replicas of tablet 087ea0527d614171b1ff3cfce4a3f776
I20260925 22:48:16.766636 13842 tablet_service.cc:1514] Processing DeleteTablet for tablet 087ea0527d614171b1ff3cfce4a3f776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:16 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:36622
I20260925 22:48:16.767138 13703 tablet_service.cc:1514] Processing DeleteTablet for tablet 087ea0527d614171b1ff3cfce4a3f776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:16 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:51446
I20260925 22:48:16.767575 13976 tablet_service.cc:1514] Processing DeleteTablet for tablet 087ea0527d614171b1ff3cfce4a3f776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:16 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:44924
I20260925 22:48:16.768854 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 13613
I20260925 22:48:16.769559 14097 tablet_replica.cc:330] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: stopping tablet replica
I20260925 22:48:16.769930 14097 raft_consensus.cc:2229] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:16.773294 14097 raft_consensus.cc:2258] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:16.775003 14097 ts_tablet_manager.cc:1905] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:16.785493 14097 ts_tablet_manager.cc:1918] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:16.785605 14097 log.cc:1196] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/087ea0527d614171b1ff3cfce4a3f776
I20260925 22:48:16.785925 14097 ts_tablet_manager.cc:1939] T 087ea0527d614171b1ff3cfce4a3f776 P b3a742219ba44019bbeb99366d13dc7f: Deleting consensus metadata
I20260925 22:48:16.787256 13562 catalog_manager.cc:4841] TS b3a742219ba44019bbeb99366d13dc7f (127.0.201.130:44773): tablet 087ea0527d614171b1ff3cfce4a3f776 (table test-table [id=e73f871d3e51456abba5a76494ce5499]) successfully deleted
I20260925 22:48:16.787699 14099 tablet_replica.cc:330] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: stopping tablet replica
I20260925 22:48:16.788008 14099 raft_consensus.cc:2229] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 1 LEADER]: Raft consensus shutting down.
W20260925 22:48:16.788285 13932 connection.cc:535] client connection to 127.0.201.129:39799 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 22:48:16.788379 13562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:39799 (error 108)
W20260925 22:48:16.788444 13932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:48:16.789058 14099 raft_consensus.cc:2258] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:16.790535 14099 ts_tablet_manager.cc:1905] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:16.793123 14099 ts_tablet_manager.cc:1918] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:16.793219 14099 log.cc:1196] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/087ea0527d614171b1ff3cfce4a3f776
I20260925 22:48:16.793529 14099 ts_tablet_manager.cc:1939] T 087ea0527d614171b1ff3cfce4a3f776 P 4ab945be52ec4586ab4a41287db05579: Deleting consensus metadata
I20260925 22:48:16.793864 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.793444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:39799 (local address 127.0.0.1:51458)
0925 22:48:16.793656 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:16.793772 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260925 22:48:16.794077 13562 catalog_manager.cc:4584] TS e2c44d00e0bd43fbb2f61db6b69ff311 (127.0.201.129:39799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 087ea0527d614171b1ff3cfce4a3f776: Network error: Client connection negotiation failed: client connection to 127.0.201.129:39799: connect: Connection refused (error 111)
I20260925 22:48:16.795394 13562 catalog_manager.cc:4841] TS 4ab945be52ec4586ab4a41287db05579 (127.0.201.131:36797): tablet 087ea0527d614171b1ff3cfce4a3f776 (table test-table [id=e73f871d3e51456abba5a76494ce5499]) successfully deleted
I20260925 22:48:16.796437 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:16.796196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:39799 (local address 127.0.201.131:53099)
0925 22:48:16.796297 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:16.796357 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 22:48:16.799969 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 13776
I20260925 22:48:16.808504 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 13912
I20260925 22:48:16.815449 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 13542
I20260925 22:48:16.824415 806 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:48:17.169433 806 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:48:17.169538 806 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:48:17.505782 806 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:48:17.506 UTC [12609] LOG: received smart shutdown request
2026-09-25 22:48:17.509 UTC [12609] LOG: background worker "logical replication launcher" (PID 12618) exited with exit code 1
2026-09-25 22:48:17.510 UTC [12613] LOG: shutting down
2026-09-25 22:48:17.675 UTC [12609] LOG: database system is shut down
2026-09-25T22:48:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48438 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:17 dist-test-slave-vdtf krb5kdc[14110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:17 dist-test-slave-vdtf krb5kdc[14110](info): set up 2 sockets
Sep 25 22:48:17 dist-test-slave-vdtf krb5kdc[14110](info): commencing operation
krb5kdc: starting...
W20260925 22:48:18.831809 806 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:48:18 dist-test-slave-vdtf krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:48:18.950785 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37803
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:37803,127.0.201.189:39385,127.0.201.188:44709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:19.057355 14123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:19.057646 14123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:19.057729 14123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:19.061722 14123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:19.061817 14123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:19.061853 14123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:19.061893 14123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:19.061928 14123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:19.067045 14123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:37803,127.0.201.189:39385,127.0.201.188:44709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:19.068106 14123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:19.069123 14123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:19.077395 14129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:19.077430 14131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.077594 14123 server_base.cc:1034] running on GCE node
W20260925 22:48:19.077764 14128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.078385 14123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:19.079481 14123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:19.080667 14123 hybrid_clock.cc:648] HybridClock initialized: now 1790376499080662 us; error 63 us; skew 500 ppm
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:19.083423 14123 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:48:19.084404 14123 webserver.cc:460] Webserver started at http://127.0.201.190:44911/ using document root <none> and password file <none>
I20260925 22:48:19.084977 14123 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:19.085071 14123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:19.085307 14123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:19.087405 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "3ad6ecd3692e4e00aedc46e5751202fa"
format_stamp: "Formatted at 2026-09-25 22:48:19 on dist-test-slave-vdtf"
server_key: "b190e36d4de4b179b3087d263743dcc6"
server_key_iv: "06f61bd61429327cc7c3fd3380ff0303"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.088006 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "3ad6ecd3692e4e00aedc46e5751202fa"
format_stamp: "Formatted at 2026-09-25 22:48:19 on dist-test-slave-vdtf"
server_key: "b190e36d4de4b179b3087d263743dcc6"
server_key_iv: "06f61bd61429327cc7c3fd3380ff0303"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.091784 14123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:48:19.094136 14138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:19.095381 14123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:48:19.095521 14123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "3ad6ecd3692e4e00aedc46e5751202fa"
format_stamp: "Formatted at 2026-09-25 22:48:19 on dist-test-slave-vdtf"
server_key: "b190e36d4de4b179b3087d263743dcc6"
server_key_iv: "06f61bd61429327cc7c3fd3380ff0303"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.095635 14123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:19.130440 14123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:19.133483 14123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:19.133697 14123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:19.140729 14123 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:37803
I20260925 22:48:19.140738 14190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:37803 every 8 connection(s)
I20260925 22:48:19.141676 14123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:19.144536 14191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:19.146637 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 14123
I20260925 22:48:19.146735 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:48:19.147042 806 external_mini_cluster.cc:1428] Setting key 9bbac94767ce9b539922570c1d69f6ec
I20260925 22:48:19.148484 14191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.201.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.189@KRBTEST.COM" created.
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:19.164367 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.150932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37803 (local address 127.0.0.1:48260)
0925 22:48:19.151396 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.151447 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:19.152293 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:19.153095 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:19.153099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:19.153410 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:19.153679 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.153691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.155400 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.155403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.156182 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.156191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.156872 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.156875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.157022 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.157654 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:19.157682 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:19.159297 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:19.161389 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.161398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.161408 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.161629 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.161956 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.161961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.161963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.162073 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.162649 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:19.162655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:19.162798 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 22:48:19.162963 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 22:48:19.163340 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":159,"thread_start_us":132,"threads_started":1}
I20260925 22:48:19.164376 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.150933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.0.201.190:37803)
0925 22:48:19.151468 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.151483 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.151537 (+ 54us) server_negotiation.cc:407] Connection header received
0925 22:48:19.152611 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.152619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.152887 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.153095 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.153828 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.155273 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.156282 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.156641 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.156910 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.159454 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.159462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.159476 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.159507 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.161257 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.161741 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.161746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.161748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.161853 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.162166 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.162171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.162172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.162535 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.162678 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.163038 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.163289 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":407,"spinlock_wait_cycles":896,"thread_start_us":129,"threads_started":1}
I20260925 22:48:19.166918 14191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:48:19.167641 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.167397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:39385 (local address 127.0.0.1:41754)
0925 22:48:19.167512 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.167584 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:39385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:48:19.167865 14139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.201.189:39385: connect: Connection refused (error 111)
I20260925 22:48:19.168610 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.168420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:39385 (local address 127.0.0.1:41768)
0925 22:48:19.168520 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.168562 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:39385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:48:19.168762 14191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.189:39385: Network error: Client connection negotiation failed: client connection to 127.0.201.189:39385: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab.
Entry for principal kudu/127.0.201.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab.
I20260925 22:48:19.172796 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.189:39385
--webserver_interface=127.0.201.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab
--principal=kudu/127.0.201.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:37803,127.0.201.189:39385,127.0.201.188:44709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:48:19.218001 14191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } attempt: 1
I20260925 22:48:19.218928 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.218594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:39385 (local address 127.0.0.1:41782)
0925 22:48:19.218763 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.218838 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:39385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 22:48:19.219619 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.219408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:39385 (local address 127.0.0.1:41784)
0925 22:48:19.219509 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.219559 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:39385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:48:19.219771 14191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.189:39385: Network error: Client connection negotiation failed: client connection to 127.0.201.189:39385: connect: Connection refused (error 111)
W20260925 22:48:19.282505 14198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:19.282748 14198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:19.282788 14198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:19.286752 14198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:19.286800 14198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:19.286820 14198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:19.286836 14198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:19.286851 14198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:19.287927 14191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } attempt: 2
I20260925 22:48:19.288740 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.288416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:39385 (local address 127.0.0.1:41794)
0925 22:48:19.288578 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.288646 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:39385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 22:48:19.289448 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.289233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:39385 (local address 127.0.0.1:41802)
0925 22:48:19.289339 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.289387 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:39385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:48:19.289595 14191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.189:39385: Network error: Client connection negotiation failed: client connection to 127.0.201.189:39385: connect: Connection refused (error 111)
I20260925 22:48:19.292506 14198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:37803,127.0.201.189:39385,127.0.201.188:44709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab
--principal=kudu/127.0.201.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.189:39385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.201.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:19.293442 14198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:19.294373 14198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:19.302709 14206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:19.302702 14203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.302778 14198 server_base.cc:1034] running on GCE node
W20260925 22:48:19.302702 14204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.303541 14198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:19.304627 14198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:19.305781 14198 hybrid_clock.cc:648] HybridClock initialized: now 1790376499305750 us; error 46 us; skew 500 ppm
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:19.308439 14198 init.cc:377] Logged in from keytab as kudu/127.0.201.189@KRBTEST.COM (short username kudu)
I20260925 22:48:19.309269 14198 webserver.cc:460] Webserver started at http://127.0.201.189:41407/ using document root <none> and password file <none>
I20260925 22:48:19.309808 14198 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:19.309906 14198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:19.310130 14198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:19.312172 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/data/instance:
uuid: "6d87190d9f7d4cfc8a46231d5a44b7db"
format_stamp: "Formatted at 2026-09-25 22:48:19 on dist-test-slave-vdtf"
server_key: "005b326285830a96ceae1a6c29187f0c"
server_key_iv: "4fd00124c6e2837ef157b5aa946b6aee"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.312711 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/wal/instance:
uuid: "6d87190d9f7d4cfc8a46231d5a44b7db"
format_stamp: "Formatted at 2026-09-25 22:48:19 on dist-test-slave-vdtf"
server_key: "005b326285830a96ceae1a6c29187f0c"
server_key_iv: "4fd00124c6e2837ef157b5aa946b6aee"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.316184 14198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:19.318452 14213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:19.319597 14198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:48:19.319757 14198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/wal
uuid: "6d87190d9f7d4cfc8a46231d5a44b7db"
format_stamp: "Formatted at 2026-09-25 22:48:19 on dist-test-slave-vdtf"
server_key: "005b326285830a96ceae1a6c29187f0c"
server_key_iv: "4fd00124c6e2837ef157b5aa946b6aee"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.319865 14198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:19.329694 14198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:19.332319 14198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:19.332520 14198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:19.339464 14198 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.189:39385
I20260925 22:48:19.339475 14265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.189:39385 every 8 connection(s)
I20260925 22:48:19.340476 14198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
I20260925 22:48:19.343255 14266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:19.346997 14266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:48:19.348362 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 14198
I20260925 22:48:19.348489 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-1/wal/instance
I20260925 22:48:19.348790 806 external_mini_cluster.cc:1428] Setting key 2a711848afa920bce484304603325526
WARNING: no policy specified for kudu/127.0.201.188@KRBTEST.COM; defaulting to no policy
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.188@KRBTEST.COM" created.
I20260925 22:48:19.361245 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.349028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.0.201.190:37803)
0925 22:48:19.349204 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.349211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.350173 (+ 962us) server_negotiation.cc:407] Connection header received
0925 22:48:19.351116 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.351120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.351185 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.351308 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.352501 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.353381 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.354485 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.354880 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.355002 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.357550 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.357554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.357576 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.357602 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.358926 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.359417 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.359422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.359424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.359508 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.359943 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.359946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.359947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.360197 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.360313 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.360813 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.360969 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:48:19.362028 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.349416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37803 (local address 127.0.0.1:48274)
0925 22:48:19.350006 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.350058 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:19.350904 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:19.351461 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:19.351468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:19.351865 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:19.352350 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.352363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.353506 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.353509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.354362 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.354371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.355020 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.355023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.355161 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.355816 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:19.355832 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:19.357394 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:19.359027 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.359031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.359043 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.359295 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.359614 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.359618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.359620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.359840 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.360311 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:19.360317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:19.360552 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 22:48:19.360701 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 22:48:19.360955 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":223,"threads_started":1}
I20260925 22:48:19.363503 14266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
I20260925 22:48:19.373922 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.364071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:39385 (local address 127.0.0.1:41816)
0925 22:48:19.364229 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.364247 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:19.364330 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:19.364841 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:19.364848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:19.365040 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:19.365174 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.365184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.366107 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.366112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.367028 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.367041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.367603 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.367606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.367759 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.368212 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:19.368223 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:19.370478 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:19.372339 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.372341 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.372342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.372521 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.372823 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.372826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.372827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.372902 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.373355 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:19.373357 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:19.373449 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 22:48:19.373556 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 22:48:19.373679 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 22:48:19.373982 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.364189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.0.201.189:39385)
0925 22:48:19.364504 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.364513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.364531 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:48:19.364564 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.364568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.364709 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.364812 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.365294 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.365982 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.367144 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.367490 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.367566 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.370618 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.370626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.370636 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.370656 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.372243 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.372632 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.372641 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.372643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.372735 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.372980 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.372982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.372983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.373266 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.373381 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.373683 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.373798 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab.
Entry for principal kudu/127.0.201.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab.
I20260925 22:48:19.376060 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.188:44709
--webserver_interface=127.0.201.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab
--principal=kudu/127.0.201.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:37803,127.0.201.189:39385,127.0.201.188:44709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:48:19.376088 14266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:48:19.376816 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.376567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35402)
0925 22:48:19.376677 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.376767 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:48:19.377012 14214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
I20260925 22:48:19.377812 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.377612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35414)
0925 22:48:19.377728 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.377768 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:48:19.378001 14266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:44709: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
I20260925 22:48:19.380734 14191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } attempt: 3
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
I20260925 22:48:19.390133 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.381222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.0.201.189:39385)
0925 22:48:19.381472 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.381475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.381484 (+ 9us) server_negotiation.cc:407] Connection header received
0925 22:48:19.381509 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.381511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.381544 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.381600 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.382689 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.383492 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.384821 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.385044 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.385157 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.387144 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.387148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.387149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.387171 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.388544 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.388957 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.388962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.388964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.389040 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.389315 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.389319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.389321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.389494 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.389596 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.389880 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.389981 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260925 22:48:19.390134 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.381153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:39385 (local address 127.0.0.1:41832)
0925 22:48:19.381253 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.381269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:19.381359 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:19.381676 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:19.381683 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:19.381909 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:19.382557 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.382571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.383620 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.383624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.384686 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.384701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.385164 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.385167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.385280 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.385817 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:19.385828 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:19.387006 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:19.388654 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.388658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.388660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.388858 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.389131 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.389135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.389137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.389219 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.389598 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:19.389602 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:19.389721 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 22:48:19.389839 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:48:19.389972 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 22:48:19.390903 14191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:48:19.391531 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.391311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35430)
0925 22:48:19.391420 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.391469 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 22:48:19.392256 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.392030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35446)
0925 22:48:19.392173 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.392213 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 22:48:19.392462 14191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:44709: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
I20260925 22:48:19.423650 14191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } attempt: 1
I20260925 22:48:19.424594 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.424303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35450)
0925 22:48:19.424453 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.424518 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 22:48:19.425294 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.425110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35464)
0925 22:48:19.425203 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.425245 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:48:19.425439 14191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:44709: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
I20260925 22:48:19.427253 14266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } attempt: 1
I20260925 22:48:19.427935 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.427647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35468)
0925 22:48:19.427805 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.427863 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 22:48:19.428614 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.428398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35470)
0925 22:48:19.428503 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.428554 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:48:19.428740 14266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:44709: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
W20260925 22:48:19.488411 14272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:19.488657 14272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:19.488705 14272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:19.492574 14272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:19.492628 14272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:19.492647 14272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:19.492664 14272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:19.492679 14272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:19.496917 14266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } attempt: 2
I20260925 22:48:19.497772 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.497470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35478)
0925 22:48:19.497628 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.497695 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 22:48:19.498320 14272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:37803,127.0.201.189:39385,127.0.201.188:44709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab
--principal=kudu/127.0.201.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.188:44709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.201.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:19.498605 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.498293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35488)
0925 22:48:19.498444 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.498506 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:48:19.498760 14266 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:44709: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
I20260925 22:48:19.499456 14272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:19.500507 14272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:48:19.500607 14191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } attempt: 2
I20260925 22:48:19.501245 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.501017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35492)
0925 22:48:19.501137 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.501183 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 22:48:19.501930 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.501725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35498)
0925 22:48:19.501836 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.501877 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:48:19.502121 14191 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:44709: Network error: Client connection negotiation failed: client connection to 127.0.201.188:44709: connect: Connection refused (error 111)
W20260925 22:48:19.509244 14281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:19.509263 14278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.509307 14272 server_base.cc:1034] running on GCE node
W20260925 22:48:19.509255 14279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.509949 14272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:19.510998 14272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:19.512161 14272 hybrid_clock.cc:648] HybridClock initialized: now 1790376499512148 us; error 24 us; skew 500 ppm
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:19.514699 14272 init.cc:377] Logged in from keytab as kudu/127.0.201.188@KRBTEST.COM (short username kudu)
I20260925 22:48:19.515522 14272 webserver.cc:460] Webserver started at http://127.0.201.188:46859/ using document root <none> and password file <none>
I20260925 22:48:19.516115 14272 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:19.516208 14272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:19.516440 14272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:19.518395 14272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/data/instance:
uuid: "fc783aa941834c568ef4db2f117a504e"
format_stamp: "Formatted at 2026-09-25 22:48:19 on dist-test-slave-vdtf"
server_key: "d14876a05132f94b788369f176499743"
server_key_iv: "7274fd5f0e094d62347601919018deda"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.518941 14272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/wal/instance:
uuid: "fc783aa941834c568ef4db2f117a504e"
format_stamp: "Formatted at 2026-09-25 22:48:19 on dist-test-slave-vdtf"
server_key: "d14876a05132f94b788369f176499743"
server_key_iv: "7274fd5f0e094d62347601919018deda"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.522764 14272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:48:19.525235 14288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:19.526427 14272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:48:19.526521 14272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/wal
uuid: "fc783aa941834c568ef4db2f117a504e"
format_stamp: "Formatted at 2026-09-25 22:48:19 on dist-test-slave-vdtf"
server_key: "d14876a05132f94b788369f176499743"
server_key_iv: "7274fd5f0e094d62347601919018deda"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.526600 14272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:19.541385 14272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:19.544169 14272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:19.544315 14272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:19.551319 14272 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.188:44709
I20260925 22:48:19.551350 14340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.188:44709 every 8 connection(s)
I20260925 22:48:19.552256 14272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
I20260925 22:48:19.552548 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 14272
I20260925 22:48:19.552672 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376341615954-806-0/minicluster-data/master-2/wal/instance
I20260925 22:48:19.553020 806 external_mini_cluster.cc:1428] Setting key fb625c8a7b18d36152a943db5c63bd69
I20260925 22:48:19.555562 14341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:19.560098 14341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:19.568053 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.554407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.0.201.188:44709)
0925 22:48:19.554827 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.554846 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.554887 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:48:19.555729 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.555752 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.556149 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.556504 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.557159 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.558635 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.559480 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.560165 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.560326 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.563068 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.563076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.563090 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.563120 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.565129 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.565582 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.565586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.565590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.565696 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.566027 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.566030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.566032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.566533 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.566739 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.567033 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.567259 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":108,"threads_started":1}
I20260925 22:48:19.571521 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.562042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.0.201.190:37803)
0925 22:48:19.562216 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.562222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.562520 (+ 298us) server_negotiation.cc:407] Connection header received
0925 22:48:19.562574 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.562577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.562631 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.562722 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.563542 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.564250 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.565424 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.565700 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.565814 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.568071 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.568074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.568076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.568097 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.569526 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.570052 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.570055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.570057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.570146 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.570448 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.570451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.570453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.570738 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.570833 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.571248 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.571342 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:48:19.571539 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.561976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37803 (local address 127.0.0.1:48290)
0925 22:48:19.562314 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.562371 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:19.562499 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:19.562781 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:19.562785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:19.563237 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:19.563421 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.563428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.564370 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.564374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.565312 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.565319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.565826 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.565830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.565923 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.566396 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:19.566409 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:19.567936 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:19.569639 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.569649 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.569655 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.569933 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.570255 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.570259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.570261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.570364 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.570852 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:19.570859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:19.570993 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 22:48:19.571165 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 22:48:19.571324 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":103,"threads_started":1}
I20260925 22:48:19.572615 14341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
I20260925 22:48:19.579111 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.570162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.0.201.190:37803)
0925 22:48:19.570523 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.570532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.570544 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:48:19.570572 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.570575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.570631 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.570720 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.571463 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.572280 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.573079 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.573447 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.573570 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.575952 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.575955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.575969 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.575986 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.577564 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.577953 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.577956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.577958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.578040 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.578315 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.578320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.578321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.578488 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.578693 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.578827 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.578935 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
I20260925 22:48:19.580595 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.573090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:39385 (local address 127.0.0.1:41842)
0925 22:48:19.573224 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.573236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:19.573300 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:19.573514 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:19.573518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:19.573685 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:19.573782 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.573787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.574506 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.574509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.575260 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.575271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.575807 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.575810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.575927 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.576331 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:19.576341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:19.577398 (+ 1057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:19.579094 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.579097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.579100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.579333 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.579646 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.579649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.579669 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.579781 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.580140 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:19.580143 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:19.580235 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 22:48:19.580352 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:48:19.580484 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 22:48:19.580672 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.573178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.0.201.189:39385)
0925 22:48:19.573292 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.573298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.573314 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:48:19.573374 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.573376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.573412 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.573476 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.573906 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.574392 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.575388 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.575665 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.575799 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.577534 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.577538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.577541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.577564 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.578966 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.579454 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.579457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.579459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.579552 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.579872 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.579876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.579878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.580046 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.580149 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.580417 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.580539 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:48:19.581329 14341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
I20260925 22:48:19.589430 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.580253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.0.201.189:39385)
0925 22:48:19.580850 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.580854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.580866 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:48:19.580898 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.580901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.580935 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.581008 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.581708 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.582401 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.583073 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.583388 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.583483 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.586122 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.586125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.586126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.586146 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.587773 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.588201 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.588203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.588204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.588271 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.588555 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.588560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.588565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.588799 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.588989 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.589175 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.589296 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":108,"threads_started":1}
I20260925 22:48:19.589918 14266 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } attempt: 3
I20260925 22:48:19.590111 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.581791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35522)
0925 22:48:19.581891 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.581905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:19.581976 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:19.582161 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:19.582163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:19.582281 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:19.582343 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.582346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.583589 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.583592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.584482 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.584496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.584935 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.584941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.585043 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.585465 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:19.585475 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:19.586716 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:19.588499 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.588502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.588523 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.588751 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.589063 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.589072 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.589074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.589183 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.589637 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:19.589641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:19.589716 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 22:48:19.589817 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:48:19.589954 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:48:19.590131 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.581860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.0.201.188:44709)
0925 22:48:19.581957 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.581961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.581973 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:48:19.582036 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.582039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.582075 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.582154 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.582441 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.583486 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.584600 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.584821 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.584900 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.586846 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.586850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.586852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.586875 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.588399 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.588874 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.588878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.588882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.588962 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.589292 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.589295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.589298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.589501 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.589604 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.589871 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.590011 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":39}
I20260925 22:48:19.593219 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:19.593782 14341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e: Bootstrap starting.
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
I20260925 22:48:19.596485 14341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:19.597425 14341 log.cc:824] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:19.599396 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.590683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35526)
0925 22:48:19.590799 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.590819 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:19.590880 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:19.591068 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:19.591069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:19.591256 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:19.591377 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.591383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.592317 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.592320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.593072 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.593082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.593649 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.593654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.593812 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.594331 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:19.594346 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:19.595871 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:19.597868 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.597870 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.597872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.598044 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.598337 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.598340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.598341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.598411 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.598914 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:19.598917 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:19.599042 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 22:48:19.599125 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:48:19.599217 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 22:48:19.599457 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.590745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.0.201.188:44709)
0925 22:48:19.590863 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.590867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.590880 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:48:19.590939 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.590942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.590981 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.591065 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.591502 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.592189 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.593211 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.593514 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.593613 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.595994 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.595998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.596000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.596023 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.597779 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.598139 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.598142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.598143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.598234 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.598569 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.598572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.598574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.598800 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.598901 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.599207 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.599330 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:48:19.601267 14191 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } attempt: 3
I20260925 22:48:19.602597 14266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db: Bootstrap starting.
I20260925 22:48:19.604904 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.595851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.0.201.189:39385)
0925 22:48:19.596028 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.596032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.596289 (+ 257us) server_negotiation.cc:407] Connection header received
0925 22:48:19.596402 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.596405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.596441 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.596535 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.597497 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.598173 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.599288 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.599593 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.599736 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.601160 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.601164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.601166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.601196 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.602879 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.603350 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.603354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.603355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.603431 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.604004 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.604008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.604010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.604199 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.604319 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.604645 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.604763 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260925 22:48:19.605515 14266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:19.606437 14266 log.cc:824] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:48:19 dist-test-slave-vdtf krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
I20260925 22:48:19.608681 14266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db: No bootstrap required, opened a new log
I20260925 22:48:19.610364 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.596505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.0.201.188:44709)
0925 22:48:19.599899 (+ 3394us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.599911 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.599924 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:48:19.599955 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.599957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.600001 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.600084 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.600931 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.601750 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.602822 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.603134 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.603226 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.604830 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.604833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.604835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.604858 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.607378 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.608103 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.608106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.608108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.608189 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.608806 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.608809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.608811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.609022 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.609150 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.610064 (+ 914us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.610177 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2880,"thread_start_us":87,"threads_started":1}
I20260925 22:48:19.610952 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.601628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:44709 (local address 127.0.0.1:35548)
0925 22:48:19.601884 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.601898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:19.601987 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:19.602546 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:19.602550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:19.602734 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:19.602856 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.602863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.603823 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.603839 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.604632 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:19.604642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.605289 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.605298 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:19.605395 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.605816 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:19.605842 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:19.607230 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:19.609436 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.609441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.609443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.609634 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.609903 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:19.609907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:19.609909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:19.610018 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:19.610462 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:19.610466 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:19.610565 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:48:19.610686 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 22:48:19.610817 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260925 22:48:19.611385 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.602142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.0.201.188:44709)
0925 22:48:19.602252 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.602257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.602270 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:48:19.602307 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.602312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.602359 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.602435 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.602990 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.603701 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.604754 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.605075 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.605170 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.607358 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.607361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.607364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.607386 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.609310 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.609728 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.609733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.609735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.609807 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.610116 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.610119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.610121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.610309 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.610412 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.611117 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.611218 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:48:19.613397 14341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e: No bootstrap required, opened a new log
I20260925 22:48:19.613864 14266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } }
I20260925 22:48:19.614073 14266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:19.614125 14266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d87190d9f7d4cfc8a46231d5a44b7db, State: Initialized, Role: FOLLOWER
I20260925 22:48:19.614645 14266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } }
I20260925 22:48:19.615108 14191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa: Bootstrap starting.
I20260925 22:48:19.616043 14266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:19.616677 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.596866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.0.201.190:37803)
0925 22:48:19.597211 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:19.597216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:19.603870 (+ 6654us) server_negotiation.cc:407] Connection header received
0925 22:48:19.605521 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:19.605528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:19.605579 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:19.605630 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:19.607738 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.608276 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.610394 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:19.610753 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:19.610829 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:19.612902 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:19.612907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:19.612912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:19.612935 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:19.614534 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.615246 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.615249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.615251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.615331 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:19.615636 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:19.615639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:19.615641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:19.615884 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:19.615984 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:19.616410 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 22:48:19.616535 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:48:19.617455 14341 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } }
I20260925 22:48:19.617734 14341 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:19.617796 14341 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc783aa941834c568ef4db2f117a504e, State: Initialized, Role: FOLLOWER
I20260925 22:48:19.617992 14191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:19.618338 14341 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } }
I20260925 22:48:19.618347 14358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } } }
I20260925 22:48:19.618487 14358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:48:19.618759 14370 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } } }
I20260925 22:48:19.618865 14370 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:48:19.618891 14191 log.cc:824] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:19.619781 14341 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:19.621032 14191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa: No bootstrap required, opened a new log
I20260925 22:48:19.619597 14198 master_runner.cc:186] Error getting master registration for 127.0.201.190:37803: OK, instance_id { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" instance_seqno: 1790376499138787 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:48:19.623466 14198 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 22:48:19.625020 14369 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:48:19.625020 14382 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:48:19.625095 14369 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 22:48:19.625238 14382 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:48:19.625262 14191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } }
I20260925 22:48:19.625479 14191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:19.625533 14191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad6ecd3692e4e00aedc46e5751202fa, State: Initialized, Role: FOLLOWER
I20260925 22:48:19.626001 14191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } }
I20260925 22:48:19.627254 14191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:19.624779 14272 master_runner.cc:186] Error getting master registration for 127.0.201.190:37803: OK, instance_id { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" instance_seqno: 1790376499138787 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:48:19.628027 14272 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:48:19.631970 14385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } } }
I20260925 22:48:19.632105 14385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [sys.catalog]: This master's current role is: FOLLOWER
W20260925 22:48:19.635298 14144 tablet.cc:2369] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:19.635406 14123 master_runner.cc:263] No leader master found from master 3ad6ecd3692e4e00aedc46e5751202fa
I20260925 22:48:19.635474 14123 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 22:48:19.635793 14396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:48:19.635859 14396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:48:19.662827 14370 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:19.663033 14370 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } }
I20260925 22:48:19.664127 14370 leader_election.cc:288] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ad6ecd3692e4e00aedc46e5751202fa (127.0.201.190:37803), 6d87190d9f7d4cfc8a46231d5a44b7db (127.0.201.189:39385)
I20260925 22:48:19.664772 14240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc783aa941834c568ef4db2f117a504e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" is_pre_election: true
I20260925 22:48:19.665069 14240 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc783aa941834c568ef4db2f117a504e in term 0.
I20260925 22:48:19.665477 14289 leader_election.cc:302] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d87190d9f7d4cfc8a46231d5a44b7db, fc783aa941834c568ef4db2f117a504e; no voters:
I20260925 22:48:19.666213 14165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc783aa941834c568ef4db2f117a504e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" is_pre_election: true
I20260925 22:48:19.666509 14165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc783aa941834c568ef4db2f117a504e in term 0.
I20260925 22:48:19.667034 14370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:19.667126 14370 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:19.667178 14370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:19.670816 14370 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } }
I20260925 22:48:19.672118 14240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc783aa941834c568ef4db2f117a504e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db"
I20260925 22:48:19.672245 14240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:19.684310 14165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc783aa941834c568ef4db2f117a504e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad6ecd3692e4e00aedc46e5751202fa"
I20260925 22:48:19.684458 14165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:19.685750 14165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc783aa941834c568ef4db2f117a504e in term 1.
I20260925 22:48:19.686093 14292 leader_election.cc:302] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad6ecd3692e4e00aedc46e5751202fa, fc783aa941834c568ef4db2f117a504e; no voters:
I20260925 22:48:19.673931 14240 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc783aa941834c568ef4db2f117a504e in term 1.
I20260925 22:48:19.686923 14370 leader_election.cc:288] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [CANDIDATE]: Term 1 election: Requested vote from peers 3ad6ecd3692e4e00aedc46e5751202fa (127.0.201.190:37803), 6d87190d9f7d4cfc8a46231d5a44b7db (127.0.201.189:39385)
I20260925 22:48:19.687071 14370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:19.687150 14370 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [term 1 LEADER]: Becoming Leader. State: Replica: fc783aa941834c568ef4db2f117a504e, State: Running, Role: LEADER
I20260925 22:48:19.687501 14370 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } }
I20260925 22:48:19.693653 14398 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc783aa941834c568ef4db2f117a504e. Latest consensus state: current_term: 1 leader_uuid: "fc783aa941834c568ef4db2f117a504e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } } }
I20260925 22:48:19.693857 14398 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:19.704310 14400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:19.705883 14400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:19.713831 14240 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [term 1 FOLLOWER]: Refusing update from remote peer fc783aa941834c568ef4db2f117a504e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:19.714550 14165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [term 1 FOLLOWER]: Refusing update from remote peer fc783aa941834c568ef4db2f117a504e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:19.715124 14398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:19.715955 14398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:19.719548 14385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc783aa941834c568ef4db2f117a504e. Latest consensus state: current_term: 1 leader_uuid: "fc783aa941834c568ef4db2f117a504e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } } }
I20260925 22:48:19.719667 14385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:48:19.723091 14358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc783aa941834c568ef4db2f117a504e. Latest consensus state: current_term: 1 leader_uuid: "fc783aa941834c568ef4db2f117a504e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } } }
I20260925 22:48:19.723212 14358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:48:19.729986 14398 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc783aa941834c568ef4db2f117a504e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } } }
I20260925 22:48:19.730103 14398 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:19.730988 14385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc783aa941834c568ef4db2f117a504e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } } }
I20260925 22:48:19.731096 14385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:48:19.731448 14401 mvcc.cc:205] Tried to move back new op lower bound from 7333382142821498880 to 7333382142739791872. Current Snapshot: MvccSnapshot[applied={T|T < 7333382142821498880}]
I20260925 22:48:19.731973 14402 mvcc.cc:205] Tried to move back new op lower bound from 7333382142821498880 to 7333382142739791872. Current Snapshot: MvccSnapshot[applied={T|T < 7333382142821498880}]
I20260925 22:48:19.735175 14370 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc783aa941834c568ef4db2f117a504e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } } }
I20260925 22:48:19.735286 14370 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:19.736263 14358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc783aa941834c568ef4db2f117a504e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6ecd3692e4e00aedc46e5751202fa" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37803 } } peers { permanent_uuid: "6d87190d9f7d4cfc8a46231d5a44b7db" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 39385 } } peers { permanent_uuid: "fc783aa941834c568ef4db2f117a504e" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 44709 } } }
I20260925 22:48:19.736383 14358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:48:19.736796 14404 mvcc.cc:205] Tried to move back new op lower bound from 7333382142821498880 to 7333382142739791872. Current Snapshot: MvccSnapshot[applied={T|T < 7333382142821498880}]
I20260925 22:48:19.737138 14400 catalog_manager.cc:1347] Generated new cluster ID: 996d7d14fcac45f6a9663ffe8cba106c
I20260925 22:48:19.737210 14400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:19.763037 14400 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:19.764264 14400 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:19.789084 14400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc783aa941834c568ef4db2f117a504e: Generated new TSK 0
I20260925 22:48:19.789922 14400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:20.626467 14369 catalog_manager.cc:1259] Loaded cluster ID: 996d7d14fcac45f6a9663ffe8cba106c
I20260925 22:48:20.626583 14369 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db: loading cluster ID for follower catalog manager: success
I20260925 22:48:20.628407 14369 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db: acquiring CA information for follower catalog manager: success
I20260925 22:48:20.629684 14369 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d87190d9f7d4cfc8a46231d5a44b7db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:48:20.637161 14396 catalog_manager.cc:1259] Loaded cluster ID: 996d7d14fcac45f6a9663ffe8cba106c
I20260925 22:48:20.637228 14396 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa: loading cluster ID for follower catalog manager: success
I20260925 22:48:20.638856 14396 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa: acquiring CA information for follower catalog manager: success
I20260925 22:48:20.639955 14396 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ad6ecd3692e4e00aedc46e5751202fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:48:20.861752 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 14123
I20260925 22:48:20.867893 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 14198
I20260925 22:48:20.873577 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 14272
2026-09-25T22:48:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3083 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:20 dist-test-slave-vdtf krb5kdc[14415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:20 dist-test-slave-vdtf krb5kdc[14415](info): set up 2 sockets
Sep 25 22:48:20 dist-test-slave-vdtf krb5kdc[14415](info): commencing operation
krb5kdc: starting...
W20260925 22:48:21.907994 806 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:48:21 dist-test-slave-vdtf krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:48:22.026258 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:33943
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:33943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:22.137499 14428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:22.137812 14428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:22.137894 14428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:22.142194 14428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:22.142277 14428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:22.142316 14428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:22.142370 14428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:22.142407 14428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:22.147775 14428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:33943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:33943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:22.148828 14428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:22.149832 14428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:22.158360 14436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:22.158515 14428 server_base.cc:1034] running on GCE node
W20260925 22:48:22.158552 14434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:22.158381 14433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:22.159188 14428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:22.160339 14428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:22.161527 14428 hybrid_clock.cc:648] HybridClock initialized: now 1790376502161515 us; error 24 us; skew 500 ppm
Sep 25 22:48:22 dist-test-slave-vdtf krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:22.164186 14428 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:48:22.165009 14428 webserver.cc:460] Webserver started at http://127.0.201.190:45365/ using document root <none> and password file <none>
I20260925 22:48:22.165560 14428 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:22.165656 14428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:22.165881 14428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:22.167908 14428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "9f950d7ac0354487b614c76fd92a6acb"
format_stamp: "Formatted at 2026-09-25 22:48:22 on dist-test-slave-vdtf"
server_key: "16de3d86c18968d13ba11903a92e693e"
server_key_iv: "d6c3135a3e1d0dc7b817b61657678ef1"
server_key_version: "encryptionkey@0"
I20260925 22:48:22.168462 14428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "9f950d7ac0354487b614c76fd92a6acb"
format_stamp: "Formatted at 2026-09-25 22:48:22 on dist-test-slave-vdtf"
server_key: "16de3d86c18968d13ba11903a92e693e"
server_key_iv: "d6c3135a3e1d0dc7b817b61657678ef1"
server_key_version: "encryptionkey@0"
I20260925 22:48:22.171970 14428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:48:22.174268 14443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:22.175539 14428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:48:22.175669 14428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "9f950d7ac0354487b614c76fd92a6acb"
format_stamp: "Formatted at 2026-09-25 22:48:22 on dist-test-slave-vdtf"
server_key: "16de3d86c18968d13ba11903a92e693e"
server_key_iv: "d6c3135a3e1d0dc7b817b61657678ef1"
server_key_version: "encryptionkey@0"
I20260925 22:48:22.175837 14428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:22.195329 14428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:22.198122 14428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:22.198333 14428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:22.205031 14428 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:33943
I20260925 22:48:22.205096 14495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:33943 every 8 connection(s)
I20260925 22:48:22.205951 14428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:22.208626 14496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:22.212863 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 14428
I20260925 22:48:22.213004 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:48:22.213274 806 external_mini_cluster.cc:1428] Setting key 3cf417aceba342fb118b332983044314
I20260925 22:48:22.214069 14496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb: Bootstrap starting.
I20260925 22:48:22.216771 14496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:22.217585 14496 log.cc:824] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:22.219498 14496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb: No bootstrap required, opened a new log
Sep 25 22:48:22 dist-test-slave-vdtf krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:22.223877 14496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f950d7ac0354487b614c76fd92a6acb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33943 } }
I20260925 22:48:22.224156 14496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:22.224257 14496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f950d7ac0354487b614c76fd92a6acb, State: Initialized, Role: FOLLOWER
I20260925 22:48:22.224762 14496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f950d7ac0354487b614c76fd92a6acb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33943 } }
I20260925 22:48:22.224884 14496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:22.224958 14496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:22.225056 14496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:22.225924 14496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f950d7ac0354487b614c76fd92a6acb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33943 } }
I20260925 22:48:22.226257 14496 leader_election.cc:302] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f950d7ac0354487b614c76fd92a6acb; no voters:
I20260925 22:48:22.226590 14496 leader_election.cc:288] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:22.226655 14501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:22.226862 14501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [term 1 LEADER]: Becoming Leader. State: Replica: 9f950d7ac0354487b614c76fd92a6acb, State: Running, Role: LEADER
I20260925 22:48:22.227195 14501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f950d7ac0354487b614c76fd92a6acb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33943 } }
I20260925 22:48:22.227870 14496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:22.228410 14502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f950d7ac0354487b614c76fd92a6acb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f950d7ac0354487b614c76fd92a6acb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33943 } } }
I20260925 22:48:22.228585 14502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:22.228441 14503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f950d7ac0354487b614c76fd92a6acb. Latest consensus state: current_term: 1 leader_uuid: "9f950d7ac0354487b614c76fd92a6acb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f950d7ac0354487b614c76fd92a6acb" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33943 } } }
I20260925 22:48:22.228505 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:22.214522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.0.201.190:33943)
0925 22:48:22.214911 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:22.214929 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:22.214968 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:48:22.215815 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:22.215837 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:22.216237 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:22.216568 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:22.217325 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:22.218710 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:22.219578 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:22.220278 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:22.220436 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:22.223143 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:22.223155 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:22.223172 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:22.223202 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:22.225310 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:22.225742 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:22.225748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:22.225753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:22.225871 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:22.226207 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:22.226210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:22.226212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:22.226686 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:22.226906 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:22.227211 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 22:48:22.227523 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":101,"threads_started":1}
I20260925 22:48:22.229218 14503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:22.229517 14507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:22.231943 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:22.232576 14507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:22.243168 14507 catalog_manager.cc:1347] Generated new cluster ID: d83908ef004b46979cc32a4c57f342fe
I20260925 22:48:22.243240 14507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:22.282801 14507 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:22.283862 14507 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:22.295022 14507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f950d7ac0354487b614c76fd92a6acb: Generated new TSK 0
I20260925 22:48:22.295881 14507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:22.339367 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 14428
2026-09-25T22:48:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1464 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:22 dist-test-slave-vdtf krb5kdc[14523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:22 dist-test-slave-vdtf krb5kdc[14523](info): set up 2 sockets
Sep 25 22:48:22 dist-test-slave-vdtf krb5kdc[14523](info): commencing operation
krb5kdc: starting...
W20260925 22:48:23.372035 806 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:48:23 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:48:23.494735 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42985
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:23.604054 14536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:23.604301 14536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:23.604342 14536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:23.608312 14536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:23.608361 14536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:23.608379 14536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:23.608438 14536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:23.608470 14536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:23.613626 14536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:42985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:23.614617 14536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:23.615592 14536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:48:23.623860 14536 server_base.cc:1034] running on GCE node
W20260925 22:48:23.623833 14542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:23.623842 14541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:23.624138 14544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:23.624720 14536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:23.625782 14536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:23.626971 14536 hybrid_clock.cc:648] HybridClock initialized: now 1790376503626944 us; error 38 us; skew 500 ppm
Sep 25 22:48:23 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:23.629676 14536 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:48:23.630575 14536 webserver.cc:460] Webserver started at http://127.0.201.190:37041/ using document root <none> and password file <none>
I20260925 22:48:23.631124 14536 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:23.631230 14536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:23.631453 14536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:23.633497 14536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "363e8357b6434ba7ace3fc57575b93c8"
format_stamp: "Formatted at 2026-09-25 22:48:23 on dist-test-slave-vdtf"
server_key: "ffdd68f83097871124f63707da7356f9"
server_key_iv: "00764007d0a1585a06c298762b499c47"
server_key_version: "encryptionkey@0"
I20260925 22:48:23.634042 14536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "363e8357b6434ba7ace3fc57575b93c8"
format_stamp: "Formatted at 2026-09-25 22:48:23 on dist-test-slave-vdtf"
server_key: "ffdd68f83097871124f63707da7356f9"
server_key_iv: "00764007d0a1585a06c298762b499c47"
server_key_version: "encryptionkey@0"
I20260925 22:48:23.637651 14536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:48:23.640081 14551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:23.641299 14536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:48:23.641399 14536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "363e8357b6434ba7ace3fc57575b93c8"
format_stamp: "Formatted at 2026-09-25 22:48:23 on dist-test-slave-vdtf"
server_key: "ffdd68f83097871124f63707da7356f9"
server_key_iv: "00764007d0a1585a06c298762b499c47"
server_key_version: "encryptionkey@0"
I20260925 22:48:23.641479 14536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:23.651990 14536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:23.654650 14536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:23.654793 14536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:23.661530 14536 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:42985
I20260925 22:48:23.661810 14603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:42985 every 8 connection(s)
I20260925 22:48:23.662475 14536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:23.665225 14604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:23.670449 14604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Bootstrap starting.
I20260925 22:48:23.670807 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 14536
I20260925 22:48:23.670930 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:48:23.671231 806 external_mini_cluster.cc:1428] Setting key d5f742d21abdad3b0edc1d2df0597cd3
I20260925 22:48:23.673115 14604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:23.673959 14604 log.cc:824] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:23.675848 14604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: No bootstrap required, opened a new log
I20260925 22:48:23.679069 14604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:23.679308 14604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:23.679415 14604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363e8357b6434ba7ace3fc57575b93c8, State: Initialized, Role: FOLLOWER
I20260925 22:48:23.679947 14604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:23.680120 14604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 25 22:48:23 dist-test-slave-vdtf krb5kdc[14523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:23.680203 14604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:23.680315 14604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:23.681356 14604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:23.681687 14604 leader_election.cc:302] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 363e8357b6434ba7ace3fc57575b93c8; no voters:
I20260925 22:48:23.682013 14604 leader_election.cc:288] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:23.682139 14609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:23.682348 14609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 1 LEADER]: Becoming Leader. State: Replica: 363e8357b6434ba7ace3fc57575b93c8, State: Running, Role: LEADER
I20260925 22:48:23.682783 14609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:23.683411 14604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:23.684250 14611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 363e8357b6434ba7ace3fc57575b93c8. Latest consensus state: current_term: 1 leader_uuid: "363e8357b6434ba7ace3fc57575b93c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } } }
I20260925 22:48:23.684430 14611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:23.684718 14610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "363e8357b6434ba7ace3fc57575b93c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } } }
I20260925 22:48:23.684825 14610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:23.686775 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:23.672585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.0.201.190:42985)
0925 22:48:23.672995 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:23.673016 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:23.673061 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:48:23.673918 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:23.673940 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:23.674371 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:23.674726 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:23.675489 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:23.676592 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:23.677455 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:23.677927 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:23.678093 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:23.680906 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:23.680911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:23.680923 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:23.680949 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:23.683146 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:23.683645 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:23.683651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:23.683656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:23.683825 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:23.684150 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:23.684155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:23.684158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:23.684685 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:23.684897 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:23.685514 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 22:48:23.685827 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":106,"threads_started":1}
W20260925 22:48:23.688654 14624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:48:23.688760 14624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:48:23.688868 14618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:23.689697 14618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:23.694802 14618 catalog_manager.cc:1347] Generated new cluster ID: a19611b52b694172bfd992c752473255
I20260925 22:48:23.694868 14618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:23.708636 14618 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:23.709560 14618 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:23.722803 14618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Generated new TSK 0
I20260925 22:48:23.723660 14618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:48:23.751579 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42985
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:23.863476 14630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:23.863788 14630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:23.863835 14630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:23.867820 14630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:23.867869 14630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:23.867956 14630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:48:23.873005 14630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:23.873994 14630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:23.875041 14630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:23.884053 14636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:23.884044 14638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:23.884097 14635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:23.884212 14630 server_base.cc:1034] running on GCE node
I20260925 22:48:23.884750 14630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:23.885375 14630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:23.886556 14630 hybrid_clock.cc:648] HybridClock initialized: now 1790376503886537 us; error 37 us; skew 500 ppm
Sep 25 22:48:23 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:23.889245 14630 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:48:23.890239 14630 webserver.cc:460] Webserver started at http://127.0.201.129:38051/ using document root <none> and password file <none>
I20260925 22:48:23.890813 14630 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:23.890906 14630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:23.891146 14630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:23.893797 14630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "f1f8f314723b4b70a1c6454a5578f265"
format_stamp: "Formatted at 2026-09-25 22:48:23 on dist-test-slave-vdtf"
server_key: "e1348c603f905fb43712b7e5e3d6eae7"
server_key_iv: "caa3e6e81493ecc492d0177a928193cf"
server_key_version: "encryptionkey@0"
I20260925 22:48:23.894544 14630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "f1f8f314723b4b70a1c6454a5578f265"
format_stamp: "Formatted at 2026-09-25 22:48:23 on dist-test-slave-vdtf"
server_key: "e1348c603f905fb43712b7e5e3d6eae7"
server_key_iv: "caa3e6e81493ecc492d0177a928193cf"
server_key_version: "encryptionkey@0"
I20260925 22:48:23.899659 14630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 22:48:23.902287 14645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:23.903481 14630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:48:23.903599 14630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "f1f8f314723b4b70a1c6454a5578f265"
format_stamp: "Formatted at 2026-09-25 22:48:23 on dist-test-slave-vdtf"
server_key: "e1348c603f905fb43712b7e5e3d6eae7"
server_key_iv: "caa3e6e81493ecc492d0177a928193cf"
server_key_version: "encryptionkey@0"
I20260925 22:48:23.903723 14630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:23.928486 14630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:23.931600 14630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:23.931864 14630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:23.932489 14630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:23.933326 14630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:23.933382 14630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:23.933462 14630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:23.933494 14630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:23.942230 14630 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:33669
I20260925 22:48:23.942255 14758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:33669 every 8 connection(s)
I20260925 22:48:23.943132 14630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:23.947829 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 14630
I20260925 22:48:23.947952 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:48:23.948252 806 external_mini_cluster.cc:1428] Setting key cb1ea64a15ba759e1d389dcfc9fcc0cd
Sep 25 22:48:23 dist-test-slave-vdtf krb5kdc[14523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:23.957916 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:23.945119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:53879 (local address 127.0.201.190:42985)
0925 22:48:23.945326 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:23.945333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:23.946557 (+ 1224us) server_negotiation.cc:407] Connection header received
0925 22:48:23.947485 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:23.947489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:23.947543 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:23.947626 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:23.949134 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:23.949773 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:23.950928 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:23.951326 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:23.951438 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:23.953991 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:23.953998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:23.954004 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:23.954038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:23.955803 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:23.956284 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:23.956289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:23.956292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:23.956389 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:23.956715 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:23.956718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:23.956720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:23.956967 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:23.957076 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:23.957566 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 22:48:23.957725 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:48:23.958693 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:23.945536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:53879)
0925 22:48:23.946402 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:23.946445 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:23.947275 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:23.947831 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:23.947839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:23.948410 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:23.948991 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:23.949008 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:23.949893 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:23.949897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:23.950801 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:23.950810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:23.951495 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:23.951498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:23.951650 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:23.952331 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:23.952348 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:23.953825 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:23.955914 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:23.955919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:23.955931 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:23.956177 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:23.956488 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:23.956490 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:23.956491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:23.956613 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:23.957079 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:23.957084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:23.957319 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 22:48:23.957471 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 22:48:23.957717 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":530,"threads_started":1}
I20260925 22:48:23.960167 14759 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42985
I20260925 22:48:23.960417 14759 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:48:23.960978 14759 heartbeater.cc:511] Master 127.0.201.190:42985 requested a full tablet report, sending...
I20260925 22:48:23.962759 14568 ts_manager.cc:194] Registered new tserver with Master: f1f8f314723b4b70a1c6454a5578f265 (127.0.201.129:33669)
I20260925 22:48:23.963953 14568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:53879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:48:23.976997 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42985
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:24.086783 14764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:24.087045 14764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:24.087088 14764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:24.091104 14764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:24.091157 14764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:24.091215 14764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:48:24.096310 14764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:24.097257 14764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:24.098204 14764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:24.107584 14772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:24.107647 14769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:24.107575 14770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:24.107753 14764 server_base.cc:1034] running on GCE node
I20260925 22:48:24.108219 14764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:24.108879 14764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:24.110065 14764 hybrid_clock.cc:648] HybridClock initialized: now 1790376504110049 us; error 28 us; skew 500 ppm
Sep 25 22:48:24 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:24.112907 14764 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:48:24.113732 14764 webserver.cc:460] Webserver started at http://127.0.201.130:39971/ using document root <none> and password file <none>
I20260925 22:48:24.114310 14764 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:24.114399 14764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:24.114634 14764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:24.116700 14764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "a617ef5412cd4b30a7c7133dc8cc34e1"
format_stamp: "Formatted at 2026-09-25 22:48:24 on dist-test-slave-vdtf"
server_key: "8451609c5150a776b7b0d1806b622b47"
server_key_iv: "82da9aeb7682b000d55d9054003048de"
server_key_version: "encryptionkey@0"
I20260925 22:48:24.117260 14764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "a617ef5412cd4b30a7c7133dc8cc34e1"
format_stamp: "Formatted at 2026-09-25 22:48:24 on dist-test-slave-vdtf"
server_key: "8451609c5150a776b7b0d1806b622b47"
server_key_iv: "82da9aeb7682b000d55d9054003048de"
server_key_version: "encryptionkey@0"
I20260925 22:48:24.120875 14764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:48:24.123189 14779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:24.124472 14764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:48:24.124612 14764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "a617ef5412cd4b30a7c7133dc8cc34e1"
format_stamp: "Formatted at 2026-09-25 22:48:24 on dist-test-slave-vdtf"
server_key: "8451609c5150a776b7b0d1806b622b47"
server_key_iv: "82da9aeb7682b000d55d9054003048de"
server_key_version: "encryptionkey@0"
I20260925 22:48:24.124727 14764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:24.142804 14764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:24.146358 14764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:24.146575 14764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:24.147184 14764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:24.148078 14764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:24.148134 14764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:24.148203 14764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:24.148237 14764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:24.157017 14764 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:44671
I20260925 22:48:24.157037 14892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:44671 every 8 connection(s)
I20260925 22:48:24.157917 14764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:24.163020 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 14764
I20260925 22:48:24.163127 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:48:24.163421 806 external_mini_cluster.cc:1428] Setting key ae7b4ab67b7a8d5c9d9afbaa4148016d
Sep 25 22:48:24 dist-test-slave-vdtf krb5kdc[14523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:48:24.172961 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.159941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:50235 (local address 127.0.201.190:42985)
0925 22:48:24.160102 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:24.160106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:24.161280 (+ 1174us) server_negotiation.cc:407] Connection header received
0925 22:48:24.162218 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:24.162221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:24.162281 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:24.162393 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:24.163770 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.164477 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.166297 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.166543 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.166674 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.169296 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:24.169300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:24.169302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:24.169328 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:24.170851 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:24.171367 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:24.171371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:24.171373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:24.171454 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:24.171865 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:24.171869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:24.171871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:24.172058 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:24.172182 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:24.172679 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 22:48:24.172798 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:48:24.173990 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.160371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:50235)
0925 22:48:24.161107 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.161152 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:24.161999 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:24.162572 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:24.162579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:24.162979 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:24.163589 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.163605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.164621 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.164625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.166122 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.166137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.166742 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.166747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.166921 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.167539 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:24.167554 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:24.169134 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:24.170975 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:24.170981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:24.170996 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:24.171263 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:24.171552 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:24.171555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:24.171557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:24.171757 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:24.172182 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:24.172190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:24.172440 (+ 250us) client_negotiation.cc:769] Sending connection context
0925 22:48:24.172625 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 22:48:24.172966 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":130,"threads_started":1}
I20260925 22:48:24.175272 14893 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42985
I20260925 22:48:24.175541 14893 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
I20260925 22:48:24.176122 14893 heartbeater.cc:511] Master 127.0.201.190:42985 requested a full tablet report, sending...
I20260925 22:48:24.177454 14568 ts_manager.cc:194] Registered new tserver with Master: a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671)
I20260925 22:48:24.178305 14568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:50235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:48:24.191203 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42985
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:24.302055 14898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:24.302325 14898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:24.302366 14898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:24.306643 14898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:24.306717 14898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:24.306820 14898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:48:24.312225 14898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:24.313266 14898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:24.314301 14898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:24.323966 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:24.324041 14898 server_base.cc:1034] running on GCE node
W20260925 22:48:24.323952 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:24.323951 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:24.324708 14898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:24.325397 14898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:24.326601 14898 hybrid_clock.cc:648] HybridClock initialized: now 1790376504326583 us; error 31 us; skew 500 ppm
Sep 25 22:48:24 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:24.329653 14898 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:48:24.330567 14898 webserver.cc:460] Webserver started at http://127.0.201.131:35117/ using document root <none> and password file <none>
I20260925 22:48:24.331132 14898 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:24.331223 14898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:24.331452 14898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:24.333514 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "9408960b3625492cb1ab22db5f5ab652"
format_stamp: "Formatted at 2026-09-25 22:48:24 on dist-test-slave-vdtf"
server_key: "3acebfb3d225b0d0036a4f84f79ceefb"
server_key_iv: "c9ff702fa79a0fd57a9d57e00b9c0f56"
server_key_version: "encryptionkey@0"
I20260925 22:48:24.334156 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "9408960b3625492cb1ab22db5f5ab652"
format_stamp: "Formatted at 2026-09-25 22:48:24 on dist-test-slave-vdtf"
server_key: "3acebfb3d225b0d0036a4f84f79ceefb"
server_key_iv: "c9ff702fa79a0fd57a9d57e00b9c0f56"
server_key_version: "encryptionkey@0"
I20260925 22:48:24.337847 14898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:48:24.340245 14913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:24.341501 14898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:48:24.341660 14898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "9408960b3625492cb1ab22db5f5ab652"
format_stamp: "Formatted at 2026-09-25 22:48:24 on dist-test-slave-vdtf"
server_key: "3acebfb3d225b0d0036a4f84f79ceefb"
server_key_iv: "c9ff702fa79a0fd57a9d57e00b9c0f56"
server_key_version: "encryptionkey@0"
I20260925 22:48:24.341774 14898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:24.366647 14898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:24.370134 14898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:24.370357 14898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:24.370978 14898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:24.371836 14898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:24.371907 14898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:24.371980 14898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:24.372023 14898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:24.381372 14898 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:40949
I20260925 22:48:24.381397 15026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:40949 every 8 connection(s)
I20260925 22:48:24.382275 14898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:24.387651 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 14898
I20260925 22:48:24.387879 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:48:24.388164 806 external_mini_cluster.cc:1428] Setting key 10e49599f80f9afa294065aeddb6c4d1
Sep 25 22:48:24 dist-test-slave-vdtf krb5kdc[14523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:24.396077 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.384244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:37201 (local address 127.0.201.190:42985)
0925 22:48:24.384467 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:24.384471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:24.385787 (+ 1316us) server_negotiation.cc:407] Connection header received
0925 22:48:24.386751 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:24.386754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:24.386816 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:24.386906 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:24.388247 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.388905 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.390048 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.390273 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.390384 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.392815 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:24.392818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:24.392820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:24.392851 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:24.394048 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:24.394542 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:24.394547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:24.394548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:24.394630 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:24.394958 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:24.394963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:24.394965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:24.395158 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:24.395265 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:24.395780 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 22:48:24.395895 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 22:48:24.396910 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.384730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:37201)
0925 22:48:24.385598 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.385643 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:24.386529 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:24.387097 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:24.387103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:24.387466 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:24.388104 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.388121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.389030 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.389033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.389927 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.389936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.390418 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.390422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.390605 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.391143 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:24.391158 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:24.392635 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:24.394172 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:24.394178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:24.394193 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:24.394441 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:24.394728 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:24.394731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:24.394733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:24.394866 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:24.395261 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:24.395268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:24.395514 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 22:48:24.395721 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 22:48:24.396057 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":116,"threads_started":1}
I20260925 22:48:24.398079 15027 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42985
I20260925 22:48:24.398330 15027 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:24.398818 15027 heartbeater.cc:511] Master 127.0.201.190:42985 requested a full tablet report, sending...
I20260925 22:48:24.399835 14568 ts_manager.cc:194] Registered new tserver with Master: 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949)
I20260925 22:48:24.400462 14568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:37201
I20260925 22:48:24.402207 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:24.411931 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.404221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.201.190:42985)
0925 22:48:24.404341 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:24.404344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:24.404533 (+ 189us) server_negotiation.cc:407] Connection header received
0925 22:48:24.404629 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:24.404633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:24.404671 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:24.404741 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:24.405438 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.406005 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.406828 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.407112 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.407224 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.408803 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:24.408807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:24.408810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:24.408837 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:24.410279 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:24.410680 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:24.410682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:24.410683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:24.410759 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:24.410981 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:24.410983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:24.410984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:24.411195 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:24.411297 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:24.411623 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 22:48:24.411803 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:48:24.415004 14568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:48:24.418082 14568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:48:24.432680 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.426429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:33669 (local address 127.0.0.1:54368)
0925 22:48:24.427565 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.427579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:24.427704 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:24.428372 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:24.428376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:24.428393 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:24.428480 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.428487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.429947 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.429950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.431189 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.431200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.432374 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.432377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.432456 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.432475 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:48:24.432527 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 22:48:24.432585 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":547,"thread_start_us":97,"threads_started":1}
I20260925 22:48:24.432826 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.426683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.0.201.129:33669)
0925 22:48:24.427365 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:24.427380 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:24.427808 (+ 428us) server_negotiation.cc:407] Connection header received
0925 22:48:24.427884 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:24.427889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:24.428060 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:24.428205 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:24.428598 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.429804 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.431351 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.432248 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.432362 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.432503 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:24.432569 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:48:24.432667 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":126,"threads_started":1}
I20260925 22:48:24.432682 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.426424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:44671 (local address 127.0.0.1:41200)
0925 22:48:24.426846 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.426899 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:24.427029 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:24.427951 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:24.427957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:24.427980 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:24.428134 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.428147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.429529 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.429535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.430892 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.430901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.432070 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.432074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.432182 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.432220 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 22:48:24.432363 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 22:48:24.432488 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":48,"thread_start_us":79,"threads_started":1}
I20260925 22:48:24.433028 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.426703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41200 (local address 127.0.201.130:44671)
0925 22:48:24.427243 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:24.427259 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:24.427285 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:48:24.427368 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:24.427375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:24.427603 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:24.427823 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:24.428272 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.429412 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.431041 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.431947 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.432069 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.432238 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:24.432754 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 22:48:24.432866 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":137,"threads_started":1}
I20260925 22:48:24.434000 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.427330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:40949 (local address 127.0.0.1:51332)
0925 22:48:24.428257 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.428268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:24.428389 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:24.428937 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:24.428940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:24.428959 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:24.429040 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.429046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.430296 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.430299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.431597 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.431608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.433611 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.433615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.433708 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.433733 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:48:24.433804 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:48:24.433873 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260925 22:48:24.434460 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.427644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.0.201.131:40949)
0925 22:48:24.428186 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:24.428197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:24.428357 (+ 160us) server_negotiation.cc:407] Connection header received
0925 22:48:24.428523 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:24.428527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:24.428716 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:24.428951 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:24.429132 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.430149 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.431806 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.432644 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.432743 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.433528 (+ 785us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:24.434221 (+ 693us) server_negotiation.cc:299] Negotiation successful
0925 22:48:24.434318 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":116,"threads_started":1}
I20260925 22:48:24.436053 14693 tablet_service.cc:1467] Processing CreateTablet for tablet aaa059d0b4c44f0ba4df32b2fac22b78 (DEFAULT_TABLE table=test-table [id=11aa75672d214fc0bca5492940ccaa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:24.437091 14693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa059d0b4c44f0ba4df32b2fac22b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:24.437016 14961 tablet_service.cc:1467] Processing CreateTablet for tablet aaa059d0b4c44f0ba4df32b2fac22b78 (DEFAULT_TABLE table=test-table [id=11aa75672d214fc0bca5492940ccaa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:24.437551 14827 tablet_service.cc:1467] Processing CreateTablet for tablet aaa059d0b4c44f0ba4df32b2fac22b78 (DEFAULT_TABLE table=test-table [id=11aa75672d214fc0bca5492940ccaa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:24.438042 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa059d0b4c44f0ba4df32b2fac22b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:24.438462 14827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa059d0b4c44f0ba4df32b2fac22b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:24.443282 15044 tablet_bootstrap.cc:492] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Bootstrap starting.
I20260925 22:48:24.445835 15046 tablet_bootstrap.cc:492] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: Bootstrap starting.
I20260925 22:48:24.446077 15044 tablet_bootstrap.cc:654] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:24.447257 15044 log.cc:824] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:24.447723 15046 tablet_bootstrap.cc:654] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:24.447960 15045 tablet_bootstrap.cc:492] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: Bootstrap starting.
I20260925 22:48:24.448921 15046 log.cc:824] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:24.449416 15044 tablet_bootstrap.cc:492] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: No bootstrap required, opened a new log
I20260925 22:48:24.449685 15044 ts_tablet_manager.cc:1397] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:48:24.450117 15045 tablet_bootstrap.cc:654] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:24.450819 15046 tablet_bootstrap.cc:492] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: No bootstrap required, opened a new log
I20260925 22:48:24.451042 15046 ts_tablet_manager.cc:1397] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:48:24.451030 15045 log.cc:824] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:24.452713 15045 tablet_bootstrap.cc:492] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: No bootstrap required, opened a new log
I20260925 22:48:24.452909 15045 ts_tablet_manager.cc:1397] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:48:24.453578 15044 raft_consensus.cc:348] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:24.453907 15044 raft_consensus.cc:374] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:24.453989 15044 raft_consensus.cc:729] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f8f314723b4b70a1c6454a5578f265, State: Initialized, Role: FOLLOWER
I20260925 22:48:24.454497 15044 consensus_queue.cc:260] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:24.455262 14759 heartbeater.cc:503] Master 127.0.201.190:42985 was elected leader, sending a full tablet report...
I20260925 22:48:24.455506 15044 ts_tablet_manager.cc:1428] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:48:24.456141 15046 raft_consensus.cc:348] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:24.456461 15046 raft_consensus.cc:374] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:24.456516 15046 raft_consensus.cc:729] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9408960b3625492cb1ab22db5f5ab652, State: Initialized, Role: FOLLOWER
I20260925 22:48:24.456670 15045 raft_consensus.cc:348] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:24.456930 15045 raft_consensus.cc:374] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:24.456991 15045 raft_consensus.cc:729] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a617ef5412cd4b30a7c7133dc8cc34e1, State: Initialized, Role: FOLLOWER
I20260925 22:48:24.456945 15046 consensus_queue.cc:260] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:24.457422 15045 consensus_queue.cc:260] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:24.457790 15046 ts_tablet_manager.cc:1428] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:48:24.458118 15027 heartbeater.cc:503] Master 127.0.201.190:42985 was elected leader, sending a full tablet report...
I20260925 22:48:24.458118 14893 heartbeater.cc:503] Master 127.0.201.190:42985 was elected leader, sending a full tablet report...
I20260925 22:48:24.458389 15045 ts_tablet_manager.cc:1428] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:48:24.475126 15051 raft_consensus.cc:482] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:24.475342 15051 raft_consensus.cc:504] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:24.476490 15051 leader_election.cc:288] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671), f1f8f314723b4b70a1c6454a5578f265 (127.0.201.129:33669)
I20260925 22:48:24.480633 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.476738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:44671 (local address 127.0.201.131:46011)
0925 22:48:24.476840 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.476857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:24.476948 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:24.477230 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:24.477234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:24.477256 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:24.477382 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.477392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.478389 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.478392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.479305 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.479314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.480198 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.480202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.480310 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.480334 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:48:24.480416 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:48:24.480485 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 22:48:24.480667 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.476820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:46011 (local address 127.0.201.130:44671)
0925 22:48:24.476943 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:24.476948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:24.476963 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:48:24.477037 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:24.477040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:24.477091 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:24.477184 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:24.477534 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.478260 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.479430 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.480076 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.480168 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.480207 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:24.480434 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 22:48:24.480485 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:48:24.481053 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.476853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:34879 (local address 127.0.201.129:33669)
0925 22:48:24.476976 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:24.476979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:24.477480 (+ 501us) server_negotiation.cc:407] Connection header received
0925 22:48:24.477625 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:24.477628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:24.477671 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:24.477755 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:24.478039 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.478751 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.479963 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.480485 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.480591 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.480646 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:24.480879 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 22:48:24.480953 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:48:24.481066 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.476738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:33669 (local address 127.0.201.131:34879)
0925 22:48:24.477371 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.477385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:24.477490 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:24.477774 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:24.477777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:24.477792 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:24.477907 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.477912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.478860 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.478863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.479849 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.479860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.480607 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.480612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.480711 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.480742 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 22:48:24.480859 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:48:24.480927 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":288,"thread_start_us":122,"threads_started":1}
I20260925 22:48:24.481304 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa059d0b4c44f0ba4df32b2fac22b78" candidate_uuid: "9408960b3625492cb1ab22db5f5ab652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" is_pre_election: true
I20260925 22:48:24.481632 14847 raft_consensus.cc:2454] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9408960b3625492cb1ab22db5f5ab652 in term 0.
I20260925 22:48:24.481879 14713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa059d0b4c44f0ba4df32b2fac22b78" candidate_uuid: "9408960b3625492cb1ab22db5f5ab652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f8f314723b4b70a1c6454a5578f265" is_pre_election: true
I20260925 22:48:24.482105 14916 leader_election.cc:302] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9408960b3625492cb1ab22db5f5ab652, a617ef5412cd4b30a7c7133dc8cc34e1; no voters:
I20260925 22:48:24.482133 14713 raft_consensus.cc:2454] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9408960b3625492cb1ab22db5f5ab652 in term 0.
I20260925 22:48:24.482388 15051 raft_consensus.cc:2781] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:24.482502 15051 raft_consensus.cc:482] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:24.482556 15051 raft_consensus.cc:3037] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:24.483578 15051 raft_consensus.cc:504] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:24.484015 15051 leader_election.cc:288] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [CANDIDATE]: Term 1 election: Requested vote from peers a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671), f1f8f314723b4b70a1c6454a5578f265 (127.0.201.129:33669)
I20260925 22:48:24.484293 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa059d0b4c44f0ba4df32b2fac22b78" candidate_uuid: "9408960b3625492cb1ab22db5f5ab652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1"
I20260925 22:48:24.484373 14713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa059d0b4c44f0ba4df32b2fac22b78" candidate_uuid: "9408960b3625492cb1ab22db5f5ab652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f8f314723b4b70a1c6454a5578f265"
I20260925 22:48:24.484433 14847 raft_consensus.cc:3037] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:24.484525 14713 raft_consensus.cc:3037] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:24.485641 14713 raft_consensus.cc:2454] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9408960b3625492cb1ab22db5f5ab652 in term 1.
I20260925 22:48:24.485648 14847 raft_consensus.cc:2454] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9408960b3625492cb1ab22db5f5ab652 in term 1.
I20260925 22:48:24.485916 14917 leader_election.cc:302] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9408960b3625492cb1ab22db5f5ab652, f1f8f314723b4b70a1c6454a5578f265; no voters:
I20260925 22:48:24.486081 15051 raft_consensus.cc:2781] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:24.486290 15051 raft_consensus.cc:686] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 1 LEADER]: Becoming Leader. State: Replica: 9408960b3625492cb1ab22db5f5ab652, State: Running, Role: LEADER
I20260925 22:48:24.486603 15051 consensus_queue.cc:237] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:24.489964 14566 catalog_manager.cc:5495] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131). New cstate: current_term: 1 leader_uuid: "9408960b3625492cb1ab22db5f5ab652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } health_report { overall_health: HEALTHY } } }
I20260925 22:48:24.515492 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.512128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.0.201.131:40949)
0925 22:48:24.512257 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:24.512261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:24.512275 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:48:24.512376 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:24.512378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:24.512431 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:24.512520 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:24.512758 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.513257 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.514069 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.514292 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.514380 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.514640 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:24.515049 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:24.515154 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:24.515224 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 22:48:24.515274 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:48:24.521366 14847 raft_consensus.cc:1261] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1 [term 1 FOLLOWER]: Refusing update from remote peer 9408960b3625492cb1ab22db5f5ab652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:24.521366 14713 raft_consensus.cc:1261] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Refusing update from remote peer 9408960b3625492cb1ab22db5f5ab652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:24.522126 15054 consensus_queue.cc:1035] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:24.522305 15051 consensus_queue.cc:1035] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:24.529644 15057 mvcc.cc:205] Tried to move back new op lower bound from 7333382162511794176 to 7333382162379923456. Current Snapshot: MvccSnapshot[applied={T|T < 7333382162511794176}]
I20260925 22:48:24.531375 15059 mvcc.cc:205] Tried to move back new op lower bound from 7333382162511794176 to 7333382162379923456. Current Snapshot: MvccSnapshot[applied={T|T < 7333382162511794176}]
I20260925 22:48:24.531921 15058 mvcc.cc:205] Tried to move back new op lower bound from 7333382162511794176 to 7333382162379923456. Current Snapshot: MvccSnapshot[applied={T|T < 7333382162511794176}]
I20260925 22:48:24.538862 14565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:24.539047 14565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:24.541467 14565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Sending DeleteTablet for 3 replicas of tablet aaa059d0b4c44f0ba4df32b2fac22b78
I20260925 22:48:24.542364 14693 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa059d0b4c44f0ba4df32b2fac22b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:24 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:54368
I20260925 22:48:24.542333 14827 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa059d0b4c44f0ba4df32b2fac22b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:24 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:41200
I20260925 22:48:24.542865 14961 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa059d0b4c44f0ba4df32b2fac22b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:24 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:51332
I20260925 22:48:24.542910 15068 tablet_replica.cc:330] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: stopping tablet replica
I20260925 22:48:24.543231 15068 raft_consensus.cc:2229] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:24.543329 15069 tablet_replica.cc:330] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: stopping tablet replica
I20260925 22:48:24.543486 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 14630
I20260925 22:48:24.543543 15068 raft_consensus.cc:2258] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:24.543936 15067 tablet_replica.cc:330] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: stopping tablet replica
I20260925 22:48:24.544199 15067 raft_consensus.cc:2229] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:24.544467 15067 raft_consensus.cc:2258] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:24.543697 15069 raft_consensus.cc:2229] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:48:24.545737 15069 raft_consensus.cc:2258] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:24.546262 15067 ts_tablet_manager.cc:1905] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:24.547211 15069 ts_tablet_manager.cc:1905] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:24.549003 15067 ts_tablet_manager.cc:1918] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:24.549103 15067 log.cc:1196] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/aaa059d0b4c44f0ba4df32b2fac22b78
I20260925 22:48:24.549428 15067 ts_tablet_manager.cc:1939] T aaa059d0b4c44f0ba4df32b2fac22b78 P a617ef5412cd4b30a7c7133dc8cc34e1: Deleting consensus metadata
I20260925 22:48:24.550740 14554 catalog_manager.cc:4841] TS a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671): tablet aaa059d0b4c44f0ba4df32b2fac22b78 (table test-table [id=11aa75672d214fc0bca5492940ccaa9e]) successfully deleted
I20260925 22:48:24.551285 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 14764
W20260925 22:48:24.551312 14555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:33669 (error 108)
I20260925 22:48:24.550838 15069 ts_tablet_manager.cc:1918] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:24.552158 15069 log.cc:1196] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/aaa059d0b4c44f0ba4df32b2fac22b78
I20260925 22:48:24.552446 15069 ts_tablet_manager.cc:1939] T aaa059d0b4c44f0ba4df32b2fac22b78 P 9408960b3625492cb1ab22db5f5ab652: Deleting consensus metadata
I20260925 22:48:24.553359 14555 catalog_manager.cc:4841] TS 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949): tablet aaa059d0b4c44f0ba4df32b2fac22b78 (table test-table [id=11aa75672d214fc0bca5492940ccaa9e]) successfully deleted
I20260925 22:48:24.556452 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.556184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:33669 (local address 127.0.0.1:54378)
0925 22:48:24.556314 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.556383 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:33669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:48:24.556643 14555 catalog_manager.cc:4584] TS f1f8f314723b4b70a1c6454a5578f265 (127.0.201.129:33669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aaa059d0b4c44f0ba4df32b2fac22b78: Network error: Client connection negotiation failed: client connection to 127.0.201.129:33669: connect: Connection refused (error 111)
I20260925 22:48:24.559562 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 14898
I20260925 22:48:24.565228 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 14536
I20260925 22:48:24.571430 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42985
--webserver_interface=127.0.201.190
--webserver_port=37041
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:24.680405 15071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:24.680666 15071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:24.680706 15071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:24.684701 15071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:24.684763 15071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:24.684783 15071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:24.684800 15071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:24.684837 15071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:24.690114 15071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:42985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=37041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:24.691192 15071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:24.692283 15071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:24.705195 15078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:24.705195 15080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:24.705211 15077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:24.705241 15071 server_base.cc:1034] running on GCE node
I20260925 22:48:24.705935 15071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:24.707166 15071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:24.708328 15071 hybrid_clock.cc:648] HybridClock initialized: now 1790376504708311 us; error 27 us; skew 500 ppm
Sep 25 22:48:24 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:24.711128 15071 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:48:24.712123 15071 webserver.cc:460] Webserver started at http://127.0.201.190:37041/ using document root <none> and password file <none>
I20260925 22:48:24.712702 15071 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:24.712826 15071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:24.716729 15071 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 22:48:24.718866 15087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:24.720132 15071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:48:24.720255 15071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "363e8357b6434ba7ace3fc57575b93c8"
format_stamp: "Formatted at 2026-09-25 22:48:23 on dist-test-slave-vdtf"
server_key: "ffdd68f83097871124f63707da7356f9"
server_key_iv: "00764007d0a1585a06c298762b499c47"
server_key_version: "encryptionkey@0"
I20260925 22:48:24.720839 15071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:24.762301 15071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:24.765513 15071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:24.765714 15071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:24.773531 15071 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:42985
I20260925 22:48:24.773857 15139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:42985 every 8 connection(s)
I20260925 22:48:24.774647 15071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:24.777808 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 15071
I20260925 22:48:24.778251 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:33669
--local_ip_for_outbound_sockets=127.0.201.129
--tserver_master_addrs=127.0.201.190:42985
--webserver_port=38051
--webserver_interface=127.0.201.129
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:48:24.779225 15140 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:48:24.780915 15140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Bootstrap starting.
I20260925 22:48:24.797653 15140 log.cc:824] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:24.806311 15140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:48:24.806684 15140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Bootstrap complete.
I20260925 22:48:24.810722 15140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:24.811138 15140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363e8357b6434ba7ace3fc57575b93c8, State: Initialized, Role: FOLLOWER
I20260925 22:48:24.811664 15140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:24.811859 15140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:24.811930 15140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:24.812022 15140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:48:24.813900 15140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:24.814239 15140 leader_election.cc:302] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 363e8357b6434ba7ace3fc57575b93c8; no voters:
I20260925 22:48:24.814599 15140 leader_election.cc:288] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:48:24.814723 15144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:48:24.814976 15144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 2 LEADER]: Becoming Leader. State: Replica: 363e8357b6434ba7ace3fc57575b93c8, State: Running, Role: LEADER
I20260925 22:48:24.815291 15144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:24.815889 15140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:24.816841 15145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "363e8357b6434ba7ace3fc57575b93c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } } }
I20260925 22:48:24.816982 15145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:24.817247 15146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 363e8357b6434ba7ace3fc57575b93c8. Latest consensus state: current_term: 2 leader_uuid: "363e8357b6434ba7ace3fc57575b93c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } } }
I20260925 22:48:24.817327 15146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:24.818004 15151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:24.822981 15151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:24.823818 15151 catalog_manager.cc:1259] Loaded cluster ID: a19611b52b694172bfd992c752473255
I20260925 22:48:24.823889 15151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:24.825388 15151 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 22:48:24.829823 15151 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 22:48:24.835451 15151 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:24.836275 15151 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Loaded TSK: 0
I20260925 22:48:24.837478 15151 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:48:24.896981 15142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:24.897271 15142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:24.897346 15142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:24.901429 15142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:24.901510 15142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:24.901614 15142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:48:24.907063 15142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:33669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=38051
--tserver_master_addrs=127.0.201.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:24.908119 15142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:24.909099 15142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:24.919655 15167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:24.919714 15168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:24.920080 15170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:24.920113 15142 server_base.cc:1034] running on GCE node
I20260925 22:48:24.920509 15142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:24.921150 15142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:24.922334 15142 hybrid_clock.cc:648] HybridClock initialized: now 1790376504922310 us; error 42 us; skew 500 ppm
Sep 25 22:48:24 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:24.925318 15142 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:48:24.926298 15142 webserver.cc:460] Webserver started at http://127.0.201.129:38051/ using document root <none> and password file <none>
I20260925 22:48:24.926877 15142 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:24.926992 15142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:24.930926 15142 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:48:24.933157 15177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:24.934464 15142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:48:24.934564 15142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "f1f8f314723b4b70a1c6454a5578f265"
format_stamp: "Formatted at 2026-09-25 22:48:23 on dist-test-slave-vdtf"
server_key: "e1348c603f905fb43712b7e5e3d6eae7"
server_key_iv: "caa3e6e81493ecc492d0177a928193cf"
server_key_version: "encryptionkey@0"
I20260925 22:48:24.935130 15142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:24.958827 15142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:24.967628 15142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:24.967849 15142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:24.968391 15142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:24.969383 15184 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:48:24.973440 15142 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:48:24.973493 15142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 22:48:24.973539 15142 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:48:24.975091 15142 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:48:24.975142 15142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:48:24.975453 15184 tablet_bootstrap.cc:492] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Bootstrap starting.
I20260925 22:48:24.984947 15142 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:33669
I20260925 22:48:24.985035 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:33669 every 8 connection(s)
I20260925 22:48:24.985894 15142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:24.986177 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 15142
I20260925 22:48:24.986713 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:44671
--local_ip_for_outbound_sockets=127.0.201.130
--tserver_master_addrs=127.0.201.190:42985
--webserver_port=39971
--webserver_interface=127.0.201.130
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:48:24.995532 15184 log.cc:824] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:48:24 dist-test-slave-vdtf krb5kdc[14523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:25.002699 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.988468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:59953 (local address 127.0.201.190:42985)
0925 22:48:24.988968 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:24.988988 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:24.990296 (+ 1308us) server_negotiation.cc:407] Connection header received
0925 22:48:24.990508 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:24.990535 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:24.990829 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:24.991209 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:24.992384 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.993335 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.994482 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.994952 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.995132 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.997781 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:24.997786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:24.997797 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:24.997823 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:24.999555 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:25.000114 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:25.000120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:25.000125 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:25.000240 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:25.000584 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:25.000587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:25.000589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:25.000915 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:25.001056 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.001624 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.001879 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":129,"threads_started":1}
I20260925 22:48:25.002749 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.988500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:59953)
0925 22:48:24.989238 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.989289 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:24.990066 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:24.991261 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:24.991270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:24.991763 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:24.992218 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.992231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.993495 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.993498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.994357 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:24.994366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:24.995098 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:24.995102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:24.995291 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:24.995958 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:24.995993 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:24.997590 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:24.999730 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:24.999736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:24.999748 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:24.999998 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:25.000338 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:25.000342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:25.000345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:25.000481 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:25.001030 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:25.001036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:25.001320 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 22:48:25.001492 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 22:48:25.001808 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":144,"threads_started":1}
I20260925 22:48:25.004236 15184 tablet_bootstrap.cc:492] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:48:25.004654 15184 tablet_bootstrap.cc:492] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Bootstrap complete.
I20260925 22:48:25.005420 15292 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42985
I20260925 22:48:25.005666 15292 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:25.006070 15184 ts_tablet_manager.cc:1397] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.007s
I20260925 22:48:25.006155 15292 heartbeater.cc:511] Master 127.0.201.190:42985 requested a full tablet report, sending...
I20260925 22:48:25.008666 15104 ts_manager.cc:194] Registered new tserver with Master: f1f8f314723b4b70a1c6454a5578f265 (127.0.201.129:33669)
I20260925 22:48:25.009047 15184 raft_consensus.cc:348] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:25.009574 15184 raft_consensus.cc:729] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f8f314723b4b70a1c6454a5578f265, State: Initialized, Role: FOLLOWER
I20260925 22:48:25.010042 15184 consensus_queue.cc:260] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } }
I20260925 22:48:25.011045 15184 ts_tablet_manager.cc:1428] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260925 22:48:25.012560 15104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:59953
I20260925 22:48:25.015028 15292 heartbeater.cc:503] Master 127.0.201.190:42985 was elected leader, sending a full tablet report...
I20260925 22:48:25.018328 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.011991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.0.201.129:33669)
0925 22:48:25.012352 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:25.012362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:25.013504 (+ 1142us) server_negotiation.cc:407] Connection header received
0925 22:48:25.013708 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:25.013713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:25.014226 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:25.014374 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:25.014673 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.015448 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.017041 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.017689 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.017809 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.017962 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.018099 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.018196 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":105,"threads_started":1}
I20260925 22:48:25.018328 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.011896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:33669 (local address 127.0.0.1:54384)
0925 22:48:25.013385 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.013417 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:25.013557 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:25.014377 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:25.014383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:25.014405 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:25.014518 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.014525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.015559 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.015562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.016917 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.016926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.017810 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.017813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.017924 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.017961 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:48:25.018073 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:48:25.018196 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"thread_start_us":99,"threads_started":1}
I20260925 22:48:25.019265 15226 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa059d0b4c44f0ba4df32b2fac22b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:24 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:54384
I20260925 22:48:25.020195 15301 tablet_replica.cc:330] stopping tablet replica
I20260925 22:48:25.020475 15301 raft_consensus.cc:2229] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:25.020727 15301 raft_consensus.cc:2258] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:25.021663 15301 ts_tablet_manager.cc:1905] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:25.025141 15301 ts_tablet_manager.cc:1918] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:25.025249 15301 log.cc:1196] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/aaa059d0b4c44f0ba4df32b2fac22b78
I20260925 22:48:25.025509 15301 ts_tablet_manager.cc:1939] T aaa059d0b4c44f0ba4df32b2fac22b78 P f1f8f314723b4b70a1c6454a5578f265: Deleting consensus metadata
I20260925 22:48:25.027083 15091 catalog_manager.cc:4841] TS f1f8f314723b4b70a1c6454a5578f265 (127.0.201.129:33669): tablet aaa059d0b4c44f0ba4df32b2fac22b78 (table test-table [id=11aa75672d214fc0bca5492940ccaa9e]) successfully deleted
W20260925 22:48:25.106253 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:25.106506 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:25.106549 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:25.110625 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:25.110689 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:25.110785 15294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:48:25.116091 15294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:44671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=39971
--tserver_master_addrs=127.0.201.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:25.117208 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:25.118292 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:48:25.127859 15294 server_base.cc:1034] running on GCE node
W20260925 22:48:25.127763 15307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:25.127763 15309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:25.127802 15306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:25.128448 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:25.129029 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:25.130208 15294 hybrid_clock.cc:648] HybridClock initialized: now 1790376505130193 us; error 25 us; skew 500 ppm
Sep 25 22:48:25 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:25.133045 15294 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:48:25.133997 15294 webserver.cc:460] Webserver started at http://127.0.201.130:39971/ using document root <none> and password file <none>
I20260925 22:48:25.134553 15294 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:25.134670 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:25.138590 15294 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260925 22:48:25.140897 15316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:25.142161 15294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:48:25.142314 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "a617ef5412cd4b30a7c7133dc8cc34e1"
format_stamp: "Formatted at 2026-09-25 22:48:24 on dist-test-slave-vdtf"
server_key: "8451609c5150a776b7b0d1806b622b47"
server_key_iv: "82da9aeb7682b000d55d9054003048de"
server_key_version: "encryptionkey@0"
I20260925 22:48:25.142863 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:25.154819 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:25.157999 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:25.158221 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:25.158820 15294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:25.159718 15294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:25.159787 15294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:25.159852 15294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:25.159904 15294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:25.168496 15294 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:44671
I20260925 22:48:25.168522 15429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:44671 every 8 connection(s)
I20260925 22:48:25.169425 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:25.174015 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 15294
I20260925 22:48:25.174407 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:40949
--local_ip_for_outbound_sockets=127.0.201.131
--tserver_master_addrs=127.0.201.190:42985
--webserver_port=35117
--webserver_interface=127.0.201.131
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:48:25 dist-test-slave-vdtf krb5kdc[14523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:25.182868 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.171173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:56177 (local address 127.0.201.190:42985)
0925 22:48:25.171317 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:25.171322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:25.172438 (+ 1116us) server_negotiation.cc:407] Connection header received
0925 22:48:25.173225 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:25.173227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:25.173274 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:25.173348 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:25.174515 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.175382 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.176565 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.176823 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.176942 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.179323 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:25.179327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:25.179329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:25.179353 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:25.180836 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:25.181345 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:25.181349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:25.181351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:25.181429 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:25.181779 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:25.181783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:25.181785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:25.181962 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:25.182070 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.182548 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.182680 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:48:25.183749 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.171610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:56177)
0925 22:48:25.172285 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.172337 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:25.173057 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:25.173508 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:25.173514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:25.173837 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:25.174363 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.174378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.175507 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.175510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.176438 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.176447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.176977 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.176981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.177151 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.177745 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:25.177761 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:25.179141 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:25.180954 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:25.180960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:25.180973 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:25.181226 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:25.181525 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:25.181529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:25.181531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:25.181671 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:25.182069 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:25.182075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:25.182321 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 22:48:25.182508 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 22:48:25.182820 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":107,"threads_started":1}
I20260925 22:48:25.184962 15430 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42985
I20260925 22:48:25.185209 15430 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:25.185753 15430 heartbeater.cc:511] Master 127.0.201.190:42985 requested a full tablet report, sending...
I20260925 22:48:25.186849 15104 ts_manager.cc:194] Registered new tserver with Master: a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671)
I20260925 22:48:25.187578 15104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:56177
W20260925 22:48:25.286770 15433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:25.287024 15433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:25.287065 15433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:25.291005 15433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:25.291059 15433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:25.291121 15433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:48:25.296299 15433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:40949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=35117
--tserver_master_addrs=127.0.201.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:25.297240 15433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:25.298192 15433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:25.307605 15439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:25.307649 15438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:25.307605 15441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:25.307947 15433 server_base.cc:1034] running on GCE node
I20260925 22:48:25.308332 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:25.308920 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:25.310120 15433 hybrid_clock.cc:648] HybridClock initialized: now 1790376505310108 us; error 30 us; skew 500 ppm
Sep 25 22:48:25 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:25.312781 15433 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:48:25.313586 15433 webserver.cc:460] Webserver started at http://127.0.201.131:35117/ using document root <none> and password file <none>
I20260925 22:48:25.314076 15433 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:25.314149 15433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:25.318107 15433 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:48:25.320144 15448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:25.321329 15433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:48:25.321416 15433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "9408960b3625492cb1ab22db5f5ab652"
format_stamp: "Formatted at 2026-09-25 22:48:24 on dist-test-slave-vdtf"
server_key: "3acebfb3d225b0d0036a4f84f79ceefb"
server_key_iv: "c9ff702fa79a0fd57a9d57e00b9c0f56"
server_key_version: "encryptionkey@0"
I20260925 22:48:25.321895 15433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:25.337575 15433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:25.340715 15433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:25.340857 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:25.341346 15433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:25.342068 15433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:25.342121 15433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:25.342159 15433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:25.342181 15433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:25.351135 15433 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:40949
I20260925 22:48:25.351145 15561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:40949 every 8 connection(s)
I20260925 22:48:25.352145 15433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:25.360798 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 15433
Sep 25 22:48:25 dist-test-slave-vdtf krb5kdc[14523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:25.365813 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.353928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:58873 (local address 127.0.201.190:42985)
0925 22:48:25.354059 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:25.354063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:25.355086 (+ 1023us) server_negotiation.cc:407] Connection header received
0925 22:48:25.355971 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:25.355974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:25.356019 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:25.356096 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:25.357173 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.357674 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.358763 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.358978 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.359134 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.361740 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:25.361744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:25.361746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:25.361781 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:25.363636 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:25.364184 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:25.364190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:25.364191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:25.364283 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:25.364631 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:25.364635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:25.364637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:25.364819 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:25.364928 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.365461 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.365608 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:48:25.366597 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.354321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:58873)
0925 22:48:25.354944 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.354982 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:25.355792 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:25.356261 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:25.356268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:25.356597 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:25.357054 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.357066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.357773 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.357776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.358622 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.358630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.359146 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.359149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.359290 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.359917 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:25.359932 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:25.361525 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:25.363791 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:25.363796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:25.363807 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:25.364071 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:25.364377 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:25.364381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:25.364383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:25.364520 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:25.364925 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:25.364932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:25.365175 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 22:48:25.365385 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 22:48:25.365729 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":112,"threads_started":1}
I20260925 22:48:25.367906 15562 heartbeater.cc:347] Connected to a master server at 127.0.201.190:42985
I20260925 22:48:25.368155 15562 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:25.368733 15562 heartbeater.cc:511] Master 127.0.201.190:42985 requested a full tablet report, sending...
I20260925 22:48:25.369681 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.361494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39918 (local address 127.0.201.190:42985)
0925 22:48:25.361723 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:25.361728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:25.361904 (+ 176us) server_negotiation.cc:407] Connection header received
0925 22:48:25.362057 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:25.362062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:25.362110 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:25.362205 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:25.362957 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.363810 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.364653 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.364888 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.365000 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.366424 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:25.366427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:25.366430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:25.366453 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:25.367996 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:25.368497 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:25.368501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:25.368503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:25.368589 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:25.368868 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:25.368871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:25.368873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:25.369064 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:25.369265 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.369407 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.369530 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260925 22:48:25.369872 15104 ts_manager.cc:194] Registered new tserver with Master: 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949)
I20260925 22:48:25.370589 15104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:58873
I20260925 22:48:25.371383 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:25.375213 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.372013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.0.201.190:42985)
0925 22:48:25.372128 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:25.372132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:25.372291 (+ 159us) server_negotiation.cc:407] Connection header received
0925 22:48:25.372397 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:25.372399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:25.372499 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:25.372603 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:25.372782 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.373259 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.374038 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.374246 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.374321 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.374603 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:25.374924 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:25.375007 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.375090 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.375133 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:48:25.375864 15103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:48:25.378957 15103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:48:25.387349 15226 tablet_service.cc:1467] Processing CreateTablet for tablet 02c2e43a12fa43e2b3b60b6a502c885a (DEFAULT_TABLE table=test-table [id=ea7b20b19fa6436bb25aa047d8b7d835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:25.388051 15226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c2e43a12fa43e2b3b60b6a502c885a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:25.390635 15184 tablet_bootstrap.cc:492] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Bootstrap starting.
I20260925 22:48:25.391435 15184 tablet_bootstrap.cc:654] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:25.392053 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.386415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:40949 (local address 127.0.0.1:51358)
0925 22:48:25.386640 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.386653 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:25.386748 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:25.387520 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:25.387523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:25.387541 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:25.387634 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.387639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.388990 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.388993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.390248 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.390259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.391582 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.391585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.391721 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.391745 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:48:25.391816 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:48:25.391909 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":28,"thread_start_us":96,"threads_started":1}
I20260925 22:48:25.392184 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.386187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.0.201.130:44671)
0925 22:48:25.386494 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:25.386508 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:25.386533 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:48:25.386623 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:25.386629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:25.386859 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:25.387045 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:25.387260 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.388647 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.390282 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.391435 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.391652 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.391830 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.391938 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.392024 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":118,"threads_started":1}
I20260925 22:48:25.392259 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.385989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:44671 (local address 127.0.0.1:41214)
0925 22:48:25.386128 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.386143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:25.386218 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:25.387007 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:25.387010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:25.387028 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:25.387159 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.387168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.388776 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.388782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.389995 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.390004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.391567 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.391570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.391787 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.391811 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:48:25.391908 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:48:25.391991 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":97}
I20260925 22:48:25.392510 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.386589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.0.201.131:40949)
0925 22:48:25.387078 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:25.387089 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:25.387112 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:48:25.387181 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:25.387186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:25.387382 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:25.387532 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:25.387841 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.388854 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.390380 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.391266 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.391375 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.391533 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.392242 (+ 709us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.392346 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":139,"threads_started":1}
I20260925 22:48:25.393921 15184 tablet_bootstrap.cc:492] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: No bootstrap required, opened a new log
I20260925 22:48:25.394012 15184 ts_tablet_manager.cc:1397] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 22:48:25.394761 15184 raft_consensus.cc:348] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:25.394891 15184 raft_consensus.cc:374] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:25.394928 15184 raft_consensus.cc:729] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f8f314723b4b70a1c6454a5578f265, State: Initialized, Role: FOLLOWER
I20260925 22:48:25.395051 15184 consensus_queue.cc:260] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:25.395310 15184 ts_tablet_manager.cc:1428] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:48:25.395320 15496 tablet_service.cc:1467] Processing CreateTablet for tablet 02c2e43a12fa43e2b3b60b6a502c885a (DEFAULT_TABLE table=test-table [id=ea7b20b19fa6436bb25aa047d8b7d835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:25.395325 15364 tablet_service.cc:1467] Processing CreateTablet for tablet 02c2e43a12fa43e2b3b60b6a502c885a (DEFAULT_TABLE table=test-table [id=ea7b20b19fa6436bb25aa047d8b7d835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:25.396389 15496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c2e43a12fa43e2b3b60b6a502c885a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:25.396401 15364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c2e43a12fa43e2b3b60b6a502c885a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:25.402995 15575 tablet_bootstrap.cc:492] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Bootstrap starting.
I20260925 22:48:25.403035 15576 tablet_bootstrap.cc:492] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: Bootstrap starting.
I20260925 22:48:25.405758 15575 tablet_bootstrap.cc:654] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:25.405762 15576 tablet_bootstrap.cc:654] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:25.406481 15576 log.cc:824] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:25.406481 15575 log.cc:824] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:25.408288 15576 tablet_bootstrap.cc:492] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: No bootstrap required, opened a new log
I20260925 22:48:25.408288 15575 tablet_bootstrap.cc:492] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: No bootstrap required, opened a new log
I20260925 22:48:25.408488 15576 ts_tablet_manager.cc:1397] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:48:25.408502 15575 ts_tablet_manager.cc:1397] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:48:25.411593 15576 raft_consensus.cc:348] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:25.411587 15575 raft_consensus.cc:348] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:25.411881 15576 raft_consensus.cc:374] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:25.411876 15575 raft_consensus.cc:374] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:25.411947 15575 raft_consensus.cc:729] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a617ef5412cd4b30a7c7133dc8cc34e1, State: Initialized, Role: FOLLOWER
I20260925 22:48:25.411962 15576 raft_consensus.cc:729] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9408960b3625492cb1ab22db5f5ab652, State: Initialized, Role: FOLLOWER
I20260925 22:48:25.412384 15576 consensus_queue.cc:260] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:25.412384 15575 consensus_queue.cc:260] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:25.413012 15430 heartbeater.cc:503] Master 127.0.201.190:42985 was elected leader, sending a full tablet report...
I20260925 22:48:25.413012 15562 heartbeater.cc:503] Master 127.0.201.190:42985 was elected leader, sending a full tablet report...
I20260925 22:48:25.414296 15576 ts_tablet_manager.cc:1428] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:48:25.414742 15575 ts_tablet_manager.cc:1428] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260925 22:48:25.420310 15431 tablet.cc:2369] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:48:25.487442 15293 tablet.cc:2369] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:48:25.603103 15563 tablet.cc:2369] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:25.663625 15580 raft_consensus.cc:482] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:25.663936 15580 raft_consensus.cc:504] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:25.665169 15580 leader_election.cc:288] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949), f1f8f314723b4b70a1c6454a5578f265 (127.0.201.129:33669)
I20260925 22:48:25.669256 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.665493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:40949 (local address 127.0.201.130:56893)
0925 22:48:25.665602 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.665618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:25.665721 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:25.666094 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:25.666097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:25.666116 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:25.666255 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.666263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.667136 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.667139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.668036 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.668045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.668800 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.668804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.668909 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.668936 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:48:25.669024 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:48:25.669102 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":1}
I20260925 22:48:25.669274 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.665598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:56893 (local address 127.0.201.131:40949)
0925 22:48:25.665747 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:25.665751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:25.665809 (+ 58us) server_negotiation.cc:407] Connection header received
0925 22:48:25.665871 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:25.665874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:25.665953 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:25.666031 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:25.666389 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.667030 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.668157 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.668676 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.668768 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.668808 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.669072 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.669141 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:48:25.669574 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.665917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:48287 (local address 127.0.201.129:33669)
0925 22:48:25.666125 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:25.666129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:25.666164 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:48:25.666284 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:25.666289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:25.666348 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:25.666434 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:25.666700 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.667257 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.668458 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.668983 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.669095 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.669140 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.669339 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.669419 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":59,"threads_started":1}
I20260925 22:48:25.669721 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.665763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:33669 (local address 127.0.201.130:48287)
0925 22:48:25.666090 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.666104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:25.666204 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:25.666454 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:25.666457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:25.666474 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:25.666581 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.666587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.667369 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.667372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.668329 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:25.668340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.669108 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.669112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:25.669203 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.669224 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:48:25.669304 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:48:25.669518 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":18,"thread_start_us":113,"threads_started":1}
I20260925 22:48:25.669987 15516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c2e43a12fa43e2b3b60b6a502c885a" candidate_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9408960b3625492cb1ab22db5f5ab652" is_pre_election: true
I20260925 22:48:25.670318 15516 raft_consensus.cc:2454] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a617ef5412cd4b30a7c7133dc8cc34e1 in term 0.
I20260925 22:48:25.670324 15246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c2e43a12fa43e2b3b60b6a502c885a" candidate_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f8f314723b4b70a1c6454a5578f265" is_pre_election: true
I20260925 22:48:25.670601 15246 raft_consensus.cc:2454] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a617ef5412cd4b30a7c7133dc8cc34e1 in term 0.
I20260925 22:48:25.670814 15320 leader_election.cc:302] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9408960b3625492cb1ab22db5f5ab652, a617ef5412cd4b30a7c7133dc8cc34e1; no voters:
I20260925 22:48:25.671111 15580 raft_consensus.cc:2781] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:25.671187 15580 raft_consensus.cc:482] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:25.671217 15580 raft_consensus.cc:3037] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:25.672344 15580 raft_consensus.cc:504] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:25.672690 15580 leader_election.cc:288] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949), f1f8f314723b4b70a1c6454a5578f265 (127.0.201.129:33669)
I20260925 22:48:25.673100 15516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c2e43a12fa43e2b3b60b6a502c885a" candidate_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9408960b3625492cb1ab22db5f5ab652"
I20260925 22:48:25.673097 15246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c2e43a12fa43e2b3b60b6a502c885a" candidate_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f8f314723b4b70a1c6454a5578f265"
I20260925 22:48:25.673220 15246 raft_consensus.cc:3037] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:25.673219 15516 raft_consensus.cc:3037] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:25.674446 15246 raft_consensus.cc:2454] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a617ef5412cd4b30a7c7133dc8cc34e1 in term 1.
I20260925 22:48:25.674446 15516 raft_consensus.cc:2454] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a617ef5412cd4b30a7c7133dc8cc34e1 in term 1.
I20260925 22:48:25.674701 15320 leader_election.cc:302] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a617ef5412cd4b30a7c7133dc8cc34e1, f1f8f314723b4b70a1c6454a5578f265; no voters:
I20260925 22:48:25.674888 15580 raft_consensus.cc:2781] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:25.675117 15580 raft_consensus.cc:686] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 1 LEADER]: Becoming Leader. State: Replica: a617ef5412cd4b30a7c7133dc8cc34e1, State: Running, Role: LEADER
I20260925 22:48:25.675446 15580 consensus_queue.cc:237] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:25.678789 15104 catalog_manager.cc:5495] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130). New cstate: current_term: 1 leader_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } health_report { overall_health: UNKNOWN } } }
I20260925 22:48:25.757033 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.753366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.0.201.130:44671)
0925 22:48:25.753496 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:25.753500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:25.753515 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:48:25.753617 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:25.753620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:25.753677 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:25.753784 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:25.754083 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.754616 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.755531 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:25.755819 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:25.755909 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:25.756273 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:25.756708 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:25.756802 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:25.756856 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 22:48:25.756905 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:48:25.762996 15516 raft_consensus.cc:1261] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652 [term 1 FOLLOWER]: Refusing update from remote peer a617ef5412cd4b30a7c7133dc8cc34e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:25.763015 15246 raft_consensus.cc:1261] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Refusing update from remote peer a617ef5412cd4b30a7c7133dc8cc34e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:25.763754 15583 consensus_queue.cc:1035] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:25.763971 15580 consensus_queue.cc:1035] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:25.770843 15587 mvcc.cc:205] Tried to move back new op lower bound from 7333382167598260224 to 7333382167249453056. Current Snapshot: MvccSnapshot[applied={T|T < 7333382167598260224}]
I20260925 22:48:25.772768 15586 mvcc.cc:205] Tried to move back new op lower bound from 7333382167598260224 to 7333382167249453056. Current Snapshot: MvccSnapshot[applied={T|T < 7333382167598260224 or (T in {7333382167598260224})}]
I20260925 22:48:25.778626 15104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:25.778786 15104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:25.780905 15104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Sending DeleteTablet for 3 replicas of tablet 02c2e43a12fa43e2b3b60b6a502c885a
I20260925 22:48:25.781526 15364 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c2e43a12fa43e2b3b60b6a502c885a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:25 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:41214
I20260925 22:48:25.782090 15226 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c2e43a12fa43e2b3b60b6a502c885a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:25 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:54384
I20260925 22:48:25.782151 15496 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c2e43a12fa43e2b3b60b6a502c885a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:25 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:51358
I20260925 22:48:25.782552 15597 tablet_replica.cc:330] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: stopping tablet replica
I20260925 22:48:25.782621 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 15142
I20260925 22:48:25.782945 15597 raft_consensus.cc:2229] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:25.783295 15597 raft_consensus.cc:2258] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:25.784874 15597 ts_tablet_manager.cc:1905] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:25.787554 15597 ts_tablet_manager.cc:1918] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:25.787683 15597 log.cc:1196] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/02c2e43a12fa43e2b3b60b6a502c885a
I20260925 22:48:25.788059 15597 ts_tablet_manager.cc:1939] T 02c2e43a12fa43e2b3b60b6a502c885a P 9408960b3625492cb1ab22db5f5ab652: Deleting consensus metadata
I20260925 22:48:25.789111 15091 catalog_manager.cc:4841] TS 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949): tablet 02c2e43a12fa43e2b3b60b6a502c885a (table test-table [id=ea7b20b19fa6436bb25aa047d8b7d835]) successfully deleted
I20260925 22:48:25.791904 15595 tablet_replica.cc:330] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: stopping tablet replica
I20260925 22:48:25.792215 15595 raft_consensus.cc:2229] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:48:25.792558 15595 raft_consensus.cc:2258] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:25.793820 15595 ts_tablet_manager.cc:1905] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:48:25.794646 15091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:33669 (error 108)
W20260925 22:48:25.795903 15299 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:48:25.795555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:33669 (local address 127.0.0.1:54398)
0925 22:48:25.795734 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.795808 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:33669: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 22:48:25.796034 15091 catalog_manager.cc:4584] TS f1f8f314723b4b70a1c6454a5578f265 (127.0.201.129:33669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02c2e43a12fa43e2b3b60b6a502c885a: Network error: Client connection negotiation failed: client connection to 127.0.201.129:33669: connect: Connection reset by peer (error 104)
I20260925 22:48:25.796099 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 15294
I20260925 22:48:25.801148 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 15433
I20260925 22:48:25.802877 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.801348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:44671 (local address 127.0.0.1:41226)
0925 22:48:25.802764 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.802810 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.130:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1341}
I20260925 22:48:25.806598 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 15071
I20260925 22:48:25.813153 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42985
--webserver_interface=127.0.201.190
--webserver_port=37041
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:25.923581 15599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:25.923879 15599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:25.923967 15599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:25.928388 15599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:25.928484 15599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:25.928524 15599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:25.928563 15599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:25.928588 15599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:25.933914 15599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:42985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=37041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:25.935004 15599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:25.936062 15599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:25.944911 15605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:25.944869 15604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:25.945068 15599 server_base.cc:1034] running on GCE node
W20260925 22:48:25.944860 15607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:25.945703 15599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:25.946753 15599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:25.947940 15599 hybrid_clock.cc:648] HybridClock initialized: now 1790376505947921 us; error 32 us; skew 500 ppm
Sep 25 22:48:25 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:25.954767 15599 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:48:25.955776 15599 webserver.cc:460] Webserver started at http://127.0.201.190:37041/ using document root <none> and password file <none>
I20260925 22:48:25.956532 15599 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:25.956612 15599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:25.960484 15599 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:48:25.962613 15614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:25.963903 15599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:48:25.964036 15599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "363e8357b6434ba7ace3fc57575b93c8"
format_stamp: "Formatted at 2026-09-25 22:48:23 on dist-test-slave-vdtf"
server_key: "ffdd68f83097871124f63707da7356f9"
server_key_iv: "00764007d0a1585a06c298762b499c47"
server_key_version: "encryptionkey@0"
I20260925 22:48:25.964655 15599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:25.986107 15599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:25.989199 15599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:25.989427 15599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:25.996441 15599 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:42985
I20260925 22:48:25.996730 15666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:42985 every 8 connection(s)
I20260925 22:48:25.997542 15599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:25.999122 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 15599
I20260925 22:48:25.999563 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:33669
--local_ip_for_outbound_sockets=127.0.201.129
--tserver_master_addrs=127.0.201.190:42985
--webserver_port=38051
--webserver_interface=127.0.201.129
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:48:26.002624 15667 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:48:26.004402 15667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Bootstrap starting.
I20260925 22:48:26.021322 15667 log.cc:824] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:26.032413 15667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:48:26.032795 15667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8: Bootstrap complete.
I20260925 22:48:26.036835 15667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:26.037200 15667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363e8357b6434ba7ace3fc57575b93c8, State: Initialized, Role: FOLLOWER
I20260925 22:48:26.037722 15667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:26.037834 15667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:26.037919 15667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:26.038033 15667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:48:26.039871 15667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:26.040185 15667 leader_election.cc:302] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 363e8357b6434ba7ace3fc57575b93c8; no voters:
I20260925 22:48:26.040505 15667 leader_election.cc:288] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:48:26.040582 15671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:48:26.040800 15671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [term 3 LEADER]: Becoming Leader. State: Replica: 363e8357b6434ba7ace3fc57575b93c8, State: Running, Role: LEADER
I20260925 22:48:26.041100 15671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } }
I20260925 22:48:26.041716 15667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:26.042034 15671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 363e8357b6434ba7ace3fc57575b93c8. Latest consensus state: current_term: 3 leader_uuid: "363e8357b6434ba7ace3fc57575b93c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } } }
I20260925 22:48:26.042158 15671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:26.042512 15675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:26.042518 15672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "363e8357b6434ba7ace3fc57575b93c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363e8357b6434ba7ace3fc57575b93c8" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42985 } } }
I20260925 22:48:26.042784 15672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363e8357b6434ba7ace3fc57575b93c8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:26.047777 15675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:26.048512 15675 catalog_manager.cc:1259] Loaded cluster ID: a19611b52b694172bfd992c752473255
I20260925 22:48:26.048591 15675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 22:48:26.049366 15675 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa1bed9a9ad google::LogMessage::Fail() at ??:0
@ 0x7fa1bed9d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa1bed9a4de google::LogMessage::Flush() at ??:0
@ 0x7fa1bed9ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa1c96d9fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa1c96dad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa1c96d7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa1c971b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa1c0fec3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa1c0fecc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa1c0fee3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa1c0fdd512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa1c26216db start_thread at ??:0
@ 0x7fa1bd89861f clone at ??:0
W20260925 22:48:26.138955 15669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:26.139205 15669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:26.139257 15669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:26.143281 15669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:26.143337 15669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:26.143393 15669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:48:26.149039 15669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:33669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=38051
--tserver_master_addrs=127.0.201.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:26.150262 15669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:26.151615 15669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:26.163041 15709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:26.163110 15708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:26.163345 15711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:26.163327 15669 server_base.cc:1034] running on GCE node
I20260925 22:48:26.163908 15669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:26.164726 15669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:26.165912 15669 hybrid_clock.cc:648] HybridClock initialized: now 1790376506165876 us; error 44 us; skew 500 ppm
Sep 25 22:48:26 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:26.169198 15669 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:48:26.170013 15669 webserver.cc:460] Webserver started at http://127.0.201.129:38051/ using document root <none> and password file <none>
I20260925 22:48:26.170584 15669 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:26.170648 15669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:26.174675 15669 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:48:26.176993 15718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:26.178422 15669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 22:48:26.178560 15669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "f1f8f314723b4b70a1c6454a5578f265"
format_stamp: "Formatted at 2026-09-25 22:48:23 on dist-test-slave-vdtf"
server_key: "e1348c603f905fb43712b7e5e3d6eae7"
server_key_iv: "caa3e6e81493ecc492d0177a928193cf"
server_key_version: "encryptionkey@0"
I20260925 22:48:26.179105 15669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:26.191934 15669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:26.195092 15669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:26.195281 15669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:26.195871 15669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:26.196941 15725 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:48:26.202157 15669 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:48:26.202240 15669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 22:48:26.202317 15669 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:48:26.204201 15669 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:48:26.204294 15669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260925 22:48:26.204586 15725 tablet_bootstrap.cc:492] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Bootstrap starting.
I20260925 22:48:26.214749 15669 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:33669
I20260925 22:48:26.214897 15832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:33669 every 8 connection(s)
I20260925 22:48:26.215667 15669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:26.216495 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 15669
I20260925 22:48:26.216925 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:44671
--local_ip_for_outbound_sockets=127.0.201.130
--tserver_master_addrs=127.0.201.190:42985
--webserver_port=39971
--webserver_interface=127.0.201.130
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:48:26.225944 15725 log.cc:824] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:26.232594 15725 tablet_bootstrap.cc:492] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:48:26.232949 15725 tablet_bootstrap.cc:492] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Bootstrap complete.
I20260925 22:48:26.234025 15725 ts_tablet_manager.cc:1397] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.009s
I20260925 22:48:26.236869 15725 raft_consensus.cc:348] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:26.237275 15725 raft_consensus.cc:729] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f8f314723b4b70a1c6454a5578f265, State: Initialized, Role: FOLLOWER
I20260925 22:48:26.237746 15725 consensus_queue.cc:260] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:26.238648 15725 ts_tablet_manager.cc:1428] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260925 22:48:26.292308 15835 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:48:26.217743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:34405)
0925 22:48:26.218299 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.218358 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:26.219456 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:26.291504 (+ 72048us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":141,"threads_started":1}
W20260925 22:48:26.292900 15833 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:42985 (0 consecutive failures): Network error: Failed to ping master at 127.0.201.190:42985: Client connection negotiation failed: client connection to 127.0.201.190:42985: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:48:26.293634 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.293377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:43899)
0925 22:48:26.293505 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.293575 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 22:48:26.294322 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.294119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:33991)
0925 22:48:26.294221 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.294274 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 22:48:26.294510 15833 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:48:26.348553 15836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:26.348831 15836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:26.348906 15836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:26.353119 15836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:26.353219 15836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:26.353309 15836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:48:26.358595 15836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:44671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=39971
--tserver_master_addrs=127.0.201.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:26.359829 15836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:26.360826 15836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:48:26.370641 15836 server_base.cc:1034] running on GCE node
W20260925 22:48:26.370589 15843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:26.370574 15846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:26.370568 15844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:26.371160 15836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:26.371812 15836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:26.372984 15836 hybrid_clock.cc:648] HybridClock initialized: now 1790376506372955 us; error 42 us; skew 500 ppm
Sep 25 22:48:26 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:26.375504 15836 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:48:26.376403 15836 webserver.cc:460] Webserver started at http://127.0.201.130:39971/ using document root <none> and password file <none>
I20260925 22:48:26.376943 15836 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:26.377032 15836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:26.380748 15836 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 22:48:26.382915 15853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:26.384168 15836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:48:26.384294 15836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "a617ef5412cd4b30a7c7133dc8cc34e1"
format_stamp: "Formatted at 2026-09-25 22:48:24 on dist-test-slave-vdtf"
server_key: "8451609c5150a776b7b0d1806b622b47"
server_key_iv: "82da9aeb7682b000d55d9054003048de"
server_key_version: "encryptionkey@0"
I20260925 22:48:26.384822 15836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:26.412452 15836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:26.415436 15836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:26.415925 15836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:26.416591 15836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:26.417737 15860 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260925 22:48:26.421614 15860 ts_tablet_manager.cc:1852] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 22:48:26.422005 15860 ts_tablet_manager.cc:1905] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:26.425778 15860 ts_tablet_manager.cc:1918] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:26.425876 15860 log.cc:1196] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/02c2e43a12fa43e2b3b60b6a502c885a
I20260925 22:48:26.426278 15860 ts_tablet_manager.cc:1939] T 02c2e43a12fa43e2b3b60b6a502c885a P a617ef5412cd4b30a7c7133dc8cc34e1: Deleting consensus metadata
I20260925 22:48:26.426601 15836 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 22:48:26.426657 15836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260925 22:48:26.426739 15836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:26.426780 15836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:26.435873 15836 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:44671
I20260925 22:48:26.435901 15967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:44671 every 8 connection(s)
I20260925 22:48:26.436810 15836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:26.440192 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.438864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:36913)
0925 22:48:26.439363 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.439477 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
W20260925 22:48:26.440696 15968 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:42985 (0 consecutive failures): Network error: Failed to ping master at 127.0.201.190:42985: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
I20260925 22:48:26.441264 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.441065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:37035)
0925 22:48:26.441168 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.441214 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 22:48:26.441823 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.441650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:33815)
0925 22:48:26.441741 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.441780 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 22:48:26.441998 15968 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:48:26.446462 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 15836
I20260925 22:48:26.446825 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:40949
--local_ip_for_outbound_sockets=127.0.201.131
--tserver_master_addrs=127.0.201.190:42985
--webserver_port=35117
--webserver_interface=127.0.201.131
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:26.558211 15971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:26.558490 15971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:26.558533 15971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:26.562557 15971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:26.562611 15971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:26.562670 15971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:48:26.568113 15971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:40949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=35117
--tserver_master_addrs=127.0.201.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:26.569156 15971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:26.570132 15971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:26.579929 15977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:26.579952 15976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:26.580152 15971 server_base.cc:1034] running on GCE node
W20260925 22:48:26.579942 15979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:26.580672 15971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:26.581336 15971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:26.582494 15971 hybrid_clock.cc:648] HybridClock initialized: now 1790376506582463 us; error 43 us; skew 500 ppm
Sep 25 22:48:26 dist-test-slave-vdtf krb5kdc[14523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:26.585397 15971 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:48:26.586295 15971 webserver.cc:460] Webserver started at http://127.0.201.131:35117/ using document root <none> and password file <none>
I20260925 22:48:26.586861 15971 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:26.586956 15971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:26.590849 15971 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:48:26.592994 15986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:26.594272 15971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:48:26.594403 15971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "9408960b3625492cb1ab22db5f5ab652"
format_stamp: "Formatted at 2026-09-25 22:48:24 on dist-test-slave-vdtf"
server_key: "3acebfb3d225b0d0036a4f84f79ceefb"
server_key_iv: "c9ff702fa79a0fd57a9d57e00b9c0f56"
server_key_version: "encryptionkey@0"
I20260925 22:48:26.594940 15971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:26.609557 15971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:26.612484 15971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:26.612726 15971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:26.613327 15971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:26.614125 15971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:26.614198 15971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:26.614291 15971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:26.614343 15971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:26.623706 15971 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:40949
I20260925 22:48:26.623741 16099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:40949 every 8 connection(s)
I20260925 22:48:26.624677 15971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:26.628258 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.626682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:36471)
0925 22:48:26.627334 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.627439 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":105,"threads_started":1}
W20260925 22:48:26.628784 16100 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:42985 (0 consecutive failures): Network error: Failed to ping master at 127.0.201.190:42985: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
I20260925 22:48:26.629396 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.629199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:48795)
0925 22:48:26.629301 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.629346 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:48:26.629994 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.629786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:46861)
0925 22:48:26.629900 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.629946 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 22:48:26.630241 16100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:48:26.633205 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 15971
I20260925 22:48:27.295462 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:27.295082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:35959)
0925 22:48:27.295303 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:27.295353 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260925 22:48:27.442924 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:27.442547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:37845)
0925 22:48:27.442781 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:27.442835 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260925 22:48:27.631121 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:27.630738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:57277)
0925 22:48:27.630974 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:27.631022 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
I20260925 22:48:27.857358 16107 raft_consensus.cc:482] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:27.857575 16107 raft_consensus.cc:504] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:27.858757 16107 leader_election.cc:288] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671), 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949)
Sep 25 22:48:27 dist-test-slave-vdtf krb5kdc[14523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.131@KRBTEST.COM
Sep 25 22:48:27 dist-test-slave-vdtf krb5kdc[14523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.130@KRBTEST.COM
I20260925 22:48:27.872486 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:27.858981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:40949 (local address 127.0.201.129:52057)
0925 22:48:27.859609 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:27.859621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:27.859777 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:27.861089 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:27.861094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:27.861630 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:27.862105 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:27.862119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:27.863360 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:27.863363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:27.864368 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:27.864377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:27.865069 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:27.865072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:27.865290 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:27.866016 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:27.866028 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:27.867594 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:27.870087 (+ 2493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:27.870093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:27.870105 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:27.870355 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:27.870711 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:27.870717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:27.870722 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:27.870853 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:27.871441 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:27.871447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:27.871715 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 22:48:27.871919 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 22:48:27.872275 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":202,"spinlock_wait_cycles":27520,"thread_start_us":59,"threads_started":1}
I20260925 22:48:27.872496 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:27.859214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:52057 (local address 127.0.201.131:40949)
0925 22:48:27.859481 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:27.859511 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:27.859734 (+ 223us) server_negotiation.cc:407] Connection header received
0925 22:48:27.860449 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:27.860454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:27.860746 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:27.861046 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:27.862265 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:27.863213 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:27.864489 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:27.864940 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:27.865114 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:27.867833 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:27.867840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:27.867855 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:27.867887 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:27.869965 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:27.870485 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:27.870498 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:27.870503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:27.870612 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:27.870954 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:27.870959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:27.870965 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:27.871323 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:27.871468 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:27.872048 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 22:48:27.872311 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260925 22:48:27.872884 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:27.859135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:42335 (local address 127.0.201.130:44671)
0925 22:48:27.859561 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:27.859577 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:27.859618 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:48:27.860405 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:27.860411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:27.860746 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:27.861044 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:27.862274 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:27.863223 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:27.864491 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:27.864941 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:27.865114 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:27.868296 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:27.868302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:27.868316 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:27.868351 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:27.870631 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:27.871065 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:27.871071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:27.871075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:27.871186 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:27.871461 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:27.871466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:27.871468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:27.871893 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:27.872023 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:27.872369 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 22:48:27.872754 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":109,"threads_started":1}
I20260925 22:48:27.872547 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:27.858941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:44671 (local address 127.0.201.129:42335)
0925 22:48:27.859376 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:27.859391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:27.859498 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:27.861089 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:27.861094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:27.861629 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:27.862106 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:27.862120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:27.863359 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:27.863364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:27.864370 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:27.864379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:27.865066 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:27.865069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:27.865290 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:27.865942 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:27.865970 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:27.868081 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:27.870746 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:27.870751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:27.870753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:27.870949 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:27.871283 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:27.871287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:27.871289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:27.871371 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:27.872013 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:27.872016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:27.872131 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 22:48:27.872233 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:48:27.872364 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":151,"spinlock_wait_cycles":2816,"thread_start_us":75,"threads_started":1}
W20260925 22:48:27.875172 15721 leader_election.cc:341] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671): Not found: Tablet not found: 02c2e43a12fa43e2b3b60b6a502c885a
W20260925 22:48:27.875664 15722 leader_election.cc:341] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949): Not found: Tablet not found: 02c2e43a12fa43e2b3b60b6a502c885a
I20260925 22:48:27.875777 15722 leader_election.cc:302] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1f8f314723b4b70a1c6454a5578f265; no voters: 9408960b3625492cb1ab22db5f5ab652, a617ef5412cd4b30a7c7133dc8cc34e1
I20260925 22:48:27.876072 16107 raft_consensus.cc:2726] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:48:28.296563 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:28.296208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:34689)
0925 22:48:28.296367 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:28.296428 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 22:48:28.444080 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:28.443639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:51721)
0925 22:48:28.443924 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:28.443976 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":124,"threads_started":1}
I20260925 22:48:28.632449 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:28.631980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:59451)
0925 22:48:28.632277 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:28.632357 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260925 22:48:29.297863 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:29.297404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:45871)
0925 22:48:29.297684 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:29.297740 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":118,"threads_started":1}
I20260925 22:48:29.445226 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:29.444813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:46773)
0925 22:48:29.445064 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:29.445133 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260925 22:48:29.465240 16116 raft_consensus.cc:482] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:29.465376 16116 raft_consensus.cc:504] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:29.465828 16116 leader_election.cc:288] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671), 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949)
W20260925 22:48:29.466712 15722 leader_election.cc:341] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949): Not found: Tablet not found: 02c2e43a12fa43e2b3b60b6a502c885a
W20260925 22:48:29.466820 15721 leader_election.cc:341] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671): Not found: Tablet not found: 02c2e43a12fa43e2b3b60b6a502c885a
I20260925 22:48:29.466936 15721 leader_election.cc:302] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1f8f314723b4b70a1c6454a5578f265; no voters: 9408960b3625492cb1ab22db5f5ab652, a617ef5412cd4b30a7c7133dc8cc34e1
I20260925 22:48:29.467087 16116 raft_consensus.cc:2726] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:48:29.633639 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:29.633242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:37301)
0925 22:48:29.633478 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:29.633538 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260925 22:48:30.299082 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:30.298692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:42703)
0925 22:48:30.298931 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:30.298987 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260925 22:48:30.446362 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:30.445962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:60469)
0925 22:48:30.446217 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:30.446275 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260925 22:48:30.634768 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:30.634374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:49289)
0925 22:48:30.634621 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:30.634677 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260925 22:48:31.300446 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:31.299992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:38855)
0925 22:48:31.300294 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:31.300352 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":77,"threads_started":1}
I20260925 22:48:31.447583 16122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:31.447170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:36751)
0925 22:48:31.447440 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:31.447491 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":106,"threads_started":1}
I20260925 22:48:31.635962 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:31.635546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:45465)
0925 22:48:31.635818 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:31.635874 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":71,"threads_started":1}
I20260925 22:48:32.301649 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:32.301209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:42869)
0925 22:48:32.301466 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:32.301527 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20260925 22:48:32.448895 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:32.448425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:44759)
0925 22:48:32.448741 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:32.448801 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":112,"threads_started":1}
I20260925 22:48:32.568005 16126 raft_consensus.cc:482] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:32.568171 16126 raft_consensus.cc:504] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:32.568614 16126 leader_election.cc:288] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671), 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949)
W20260925 22:48:32.569409 15722 leader_election.cc:341] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949): Not found: Tablet not found: 02c2e43a12fa43e2b3b60b6a502c885a
W20260925 22:48:32.569521 15721 leader_election.cc:341] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671): Not found: Tablet not found: 02c2e43a12fa43e2b3b60b6a502c885a
I20260925 22:48:32.569636 15721 leader_election.cc:302] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1f8f314723b4b70a1c6454a5578f265; no voters: 9408960b3625492cb1ab22db5f5ab652, a617ef5412cd4b30a7c7133dc8cc34e1
I20260925 22:48:32.569815 16126 raft_consensus.cc:2726] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:48:32.637097 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:32.636707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:43195)
0925 22:48:32.636946 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:32.637004 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260925 22:48:33.302994 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:33.302500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:46123)
0925 22:48:33.302775 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:33.302887 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260925 22:48:33.450301 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:33.449769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:45291)
0925 22:48:33.450119 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:33.450195 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":69,"threads_started":1}
I20260925 22:48:33.638255 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:33.637823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:60949)
0925 22:48:33.638112 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:33.638167 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260925 22:48:34.304278 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:34.303835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:57155)
0925 22:48:34.304122 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:34.304181 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":103,"threads_started":1}
I20260925 22:48:34.451476 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:34.451120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:60759)
0925 22:48:34.451338 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:34.451387 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260925 22:48:34.639420 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:34.638985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:50595)
0925 22:48:34.639270 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:34.639327 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260925 22:48:35.305405 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:35.304980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:58855)
0925 22:48:35.305239 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:35.305307 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260925 22:48:35.452582 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:35.452160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:43285)
0925 22:48:35.452433 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:35.452491 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260925 22:48:35.640642 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:35.640248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:41875)
0925 22:48:35.640485 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:35.640541 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260925 22:48:36.306764 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.306359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:43135)
0925 22:48:36.306603 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.306665 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260925 22:48:36.453739 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.453350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:54037)
0925 22:48:36.453589 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.453643 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
I20260925 22:48:36.641842 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.641415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:43493)
0925 22:48:36.641691 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.641749 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260925 22:48:37.307960 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.307535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:43695)
0925 22:48:37.307811 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:37.307864 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":72,"threads_started":1}
I20260925 22:48:37.454840 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.454466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:57583)
0925 22:48:37.454697 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:37.454748 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
I20260925 22:48:37.510447 16142 raft_consensus.cc:482] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:37.510571 16142 raft_consensus.cc:504] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a617ef5412cd4b30a7c7133dc8cc34e1" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 44671 } } peers { permanent_uuid: "9408960b3625492cb1ab22db5f5ab652" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 40949 } } peers { permanent_uuid: "f1f8f314723b4b70a1c6454a5578f265" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 33669 } }
I20260925 22:48:37.511039 16142 leader_election.cc:288] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671), 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949)
W20260925 22:48:37.511778 15721 leader_election.cc:341] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a617ef5412cd4b30a7c7133dc8cc34e1 (127.0.201.130:44671): Not found: Tablet not found: 02c2e43a12fa43e2b3b60b6a502c885a
W20260925 22:48:37.511902 15722 leader_election.cc:341] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9408960b3625492cb1ab22db5f5ab652 (127.0.201.131:40949): Not found: Tablet not found: 02c2e43a12fa43e2b3b60b6a502c885a
I20260925 22:48:37.512019 15721 leader_election.cc:302] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1f8f314723b4b70a1c6454a5578f265; no voters: 9408960b3625492cb1ab22db5f5ab652, a617ef5412cd4b30a7c7133dc8cc34e1
I20260925 22:48:37.512156 16142 raft_consensus.cc:2726] T 02c2e43a12fa43e2b3b60b6a502c885a P f1f8f314723b4b70a1c6454a5578f265 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:48:37.643002 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.642580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:33349)
0925 22:48:37.642832 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:37.642885 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
I20260925 22:48:38.309149 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.308662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:36497)
0925 22:48:38.308951 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:38.309045 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":93,"threads_started":1}
I20260925 22:48:38.456033 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.455572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:36081)
0925 22:48:38.455866 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:38.455924 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":140,"threads_started":1}
I20260925 22:48:38.644382 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.643912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:35969)
0925 22:48:38.644196 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:38.644255 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260925 22:48:39.310315 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:39.309913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:52523)
0925 22:48:39.310152 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:39.310210 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260925 22:48:39.457259 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:39.456791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:39201)
0925 22:48:39.457101 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:39.457162 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":95,"threads_started":1}
I20260925 22:48:39.645638 16149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:39.645187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:39811)
0925 22:48:39.645480 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:39.645542 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260925 22:48:40.311441 16150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.311054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:33737)
0925 22:48:40.311289 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.311342 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260925 22:48:40.458388 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.457991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:53413)
0925 22:48:40.458244 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.458300 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260925 22:48:40.646778 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.646361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:57135)
0925 22:48:40.646624 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.646684 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260925 22:48:41.312695 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.312291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.129:39663)
0925 22:48:41.312542 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.312597 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260925 22:48:41.459580 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.459203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.130:49489)
0925 22:48:41.459430 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.459488 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
I20260925 22:48:41.641436 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 15669
I20260925 22:48:41.647220 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 15836
I20260925 22:48:41.649317 16155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.648832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:42985 (local address 127.0.201.131:53961)
0925 22:48:41.649107 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.649180 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260925 22:48:41.652782 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 15971
2026-09-25T22:48:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19314 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:41 dist-test-slave-vdtf krb5kdc[16158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:41 dist-test-slave-vdtf krb5kdc[16158](info): set up 2 sockets
Sep 25 22:48:41 dist-test-slave-vdtf krb5kdc[16158](info): commencing operation
krb5kdc: starting...
W20260925 22:48:42.699977 806 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:48:42 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:48:42.820740 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37517
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:37517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:42.929930 16171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:42.930191 16171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:42.930230 16171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:42.934270 16171 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:48:42.934329 16171 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:48:42.934345 16171 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:48:42.934360 16171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:42.934377 16171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:42.934434 16171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:42.934468 16171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:42.934492 16171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:42.939790 16171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:37517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:42.940815 16171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:42.941833 16171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:48:42.950590 16171 server_base.cc:1034] running on GCE node
W20260925 22:48:42.950546 16176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:42.950541 16179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:42.950562 16177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:42.951426 16171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:42.952540 16171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:42.953738 16171 hybrid_clock.cc:648] HybridClock initialized: now 1790376522953722 us; error 34 us; skew 500 ppm
Sep 25 22:48:42 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:42.962761 16171 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:48:42.963760 16171 webserver.cc:460] Webserver started at http://127.0.201.190:46511/ using document root <none> and password file <none>
I20260925 22:48:42.964335 16171 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:42.964427 16171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:42.964656 16171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:42.966737 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "1facd2cfc739458db9d7a62a276c0fef"
format_stamp: "Formatted at 2026-09-25 22:48:42 on dist-test-slave-vdtf"
server_key: "6bf12e35920326c8c0abf84718575c46"
server_key_iv: "1c324b65749d926b7008eeb612f38b4e"
server_key_version: "encryptionkey@0"
I20260925 22:48:42.967289 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "1facd2cfc739458db9d7a62a276c0fef"
format_stamp: "Formatted at 2026-09-25 22:48:42 on dist-test-slave-vdtf"
server_key: "6bf12e35920326c8c0abf84718575c46"
server_key_iv: "1c324b65749d926b7008eeb612f38b4e"
server_key_version: "encryptionkey@0"
I20260925 22:48:42.970949 16171 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:48:42.973310 16186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:42.974494 16171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:48:42.974627 16171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "1facd2cfc739458db9d7a62a276c0fef"
format_stamp: "Formatted at 2026-09-25 22:48:42 on dist-test-slave-vdtf"
server_key: "6bf12e35920326c8c0abf84718575c46"
server_key_iv: "1c324b65749d926b7008eeb612f38b4e"
server_key_version: "encryptionkey@0"
I20260925 22:48:42.974731 16171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:42.989356 16171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:42.992131 16171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:42.992319 16171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:42.999028 16171 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:37517
I20260925 22:48:42.999361 16238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:37517 every 8 connection(s)
I20260925 22:48:43.000063 16171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:43.002727 16239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:43.006910 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 16171
I20260925 22:48:43.007030 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:48:43.007351 806 external_mini_cluster.cc:1428] Setting key 41db041fb8290ce2ea81d26d327d766c
I20260925 22:48:43.008635 16239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Bootstrap starting.
I20260925 22:48:43.010989 16239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:43.011847 16239 log.cc:824] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:43.013865 16239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: No bootstrap required, opened a new log
Sep 25 22:48:43 dist-test-slave-vdtf krb5kdc[16158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:43.018613 16239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:48:43.018894 16239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:43.018956 16239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1facd2cfc739458db9d7a62a276c0fef, State: Initialized, Role: FOLLOWER
I20260925 22:48:43.019392 16239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:48:43.019564 16239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:43.019639 16239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:43.019766 16239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:43.020884 16239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:48:43.021239 16239 leader_election.cc:302] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1facd2cfc739458db9d7a62a276c0fef; no voters:
I20260925 22:48:43.021550 16239 leader_election.cc:288] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:43.021646 16244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:43.021895 16244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 1 LEADER]: Becoming Leader. State: Replica: 1facd2cfc739458db9d7a62a276c0fef, State: Running, Role: LEADER
I20260925 22:48:43.022223 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.008746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.0.201.190:37517)
0925 22:48:43.009175 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.009190 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.009225 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:48:43.009850 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.009865 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.010180 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.010468 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:43.011393 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.012462 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.013567 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.014054 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.014168 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.016916 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:43.016924 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:43.016944 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:43.016983 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:43.018980 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.019492 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.019498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.019503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.019646 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.019984 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.019990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.019994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.020495 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:43.020696 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.020926 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.021227 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":152,"threads_started":1}
I20260925 22:48:43.022243 16244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:48:43.022821 16239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:43.023378 16244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1facd2cfc739458db9d7a62a276c0fef. Latest consensus state: current_term: 1 leader_uuid: "1facd2cfc739458db9d7a62a276c0fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } } }
I20260925 22:48:43.023592 16244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:43.023826 16245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1facd2cfc739458db9d7a62a276c0fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } } }
I20260925 22:48:43.023939 16245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:43.024449 16253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:43.027611 16253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:43.033289 16253 catalog_manager.cc:1347] Generated new cluster ID: 8e98aab1064d463696c76819421d324d
I20260925 22:48:43.033358 16253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:43.044934 16253 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:43.045863 16253 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:43.050312 16253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Generated new TSK 0
I20260925 22:48:43.051061 16253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:48:43.091176 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37517
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:43.208794 16265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:43.209029 16265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:43.209070 16265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:43.212972 16265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:43.213017 16265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:43.213073 16265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:48:43.218305 16265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:43.219249 16265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:43.220221 16265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:43.229494 16273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:43.229491 16270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.229580 16265 server_base.cc:1034] running on GCE node
W20260925 22:48:43.229491 16271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.230122 16265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:43.230698 16265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:43.231843 16265 hybrid_clock.cc:648] HybridClock initialized: now 1790376523231830 us; error 24 us; skew 500 ppm
Sep 25 22:48:43 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:43.234382 16265 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:48:43.235160 16265 webserver.cc:460] Webserver started at http://127.0.201.129:36003/ using document root <none> and password file <none>
I20260925 22:48:43.235749 16265 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:43.235837 16265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:43.236056 16265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:43.238034 16265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "028d8019ade7490c8229a79f84f0bf29"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "bb603a8e20f77fea6234ca7a16931224"
server_key_iv: "8270cc92bf22e150afd22fa0fd83225b"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.238580 16265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "028d8019ade7490c8229a79f84f0bf29"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "bb603a8e20f77fea6234ca7a16931224"
server_key_iv: "8270cc92bf22e150afd22fa0fd83225b"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.242144 16265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:48:43.244620 16280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.245796 16265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:48:43.245927 16265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "028d8019ade7490c8229a79f84f0bf29"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "bb603a8e20f77fea6234ca7a16931224"
server_key_iv: "8270cc92bf22e150afd22fa0fd83225b"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.246037 16265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:43.259127 16265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:43.261768 16265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:43.261955 16265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:43.262490 16265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:43.263271 16265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:43.263338 16265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.263406 16265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:43.263447 16265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.271935 16265 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:37303
I20260925 22:48:43.271970 16393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:37303 every 8 connection(s)
I20260925 22:48:43.272781 16265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:43.277765 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 16265
I20260925 22:48:43.277917 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:48:43.278215 806 external_mini_cluster.cc:1428] Setting key 914a10a40add55c0481ee0503cb9380e
Sep 25 22:48:43 dist-test-slave-vdtf krb5kdc[16158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:43.286861 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.274509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:59835 (local address 127.0.201.190:37517)
0925 22:48:43.274639 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.274642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.275594 (+ 952us) server_negotiation.cc:407] Connection header received
0925 22:48:43.276515 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.276519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.276577 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.276674 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:43.277866 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.278643 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.280109 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.280338 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.280419 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.282865 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:43.282871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:43.282878 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:43.282910 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:43.284664 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.285220 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.285226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.285231 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.285327 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.285690 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.285695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.285704 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.285957 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:43.286067 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.286537 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.286670 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:48:43.287534 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.274843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:59835)
0925 22:48:43.275457 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.275496 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.276313 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.276844 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.276850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.277210 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:43.277731 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.277745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.278792 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.278795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.279993 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.280006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.280483 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.280490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.280724 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.281339 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:43.281363 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:43.282713 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:43.284780 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.284785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.284800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.285114 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.285435 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.285438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.285440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.285595 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.286078 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:43.286083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:43.286296 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.286451 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.286689 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:48:43.288702 16394 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37517
I20260925 22:48:43.288913 16394 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:43.289392 16394 heartbeater.cc:511] Master 127.0.201.190:37517 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:48:43.290855 16203 ts_manager.cc:194] Registered new tserver with Master: 028d8019ade7490c8229a79f84f0bf29 (127.0.201.129:37303)
I20260925 22:48:43.292071 16203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:59835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:48:43.305866 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37517
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:43.414878 16399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:43.415113 16399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:43.415154 16399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:43.419003 16399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:43.419051 16399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:43.419106 16399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:48:43.424283 16399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:43.425237 16399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:43.426359 16399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:43.435667 16404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:43.435633 16407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.435822 16399 server_base.cc:1034] running on GCE node
W20260925 22:48:43.435645 16405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.436255 16399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:43.436851 16399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:43.438023 16399 hybrid_clock.cc:648] HybridClock initialized: now 1790376523438010 us; error 24 us; skew 500 ppm
Sep 25 22:48:43 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:43.440582 16399 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:48:43.441391 16399 webserver.cc:460] Webserver started at http://127.0.201.130:45939/ using document root <none> and password file <none>
I20260925 22:48:43.441926 16399 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:43.442018 16399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:43.442231 16399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:43.444213 16399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "d9c8f08ebc7d406c952e5ee8295bfa52"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "5a46cf996036a979802b363263021244"
server_key_iv: "f1c9d1799a8bee0cdeabb66ec7184a43"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.444741 16399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "d9c8f08ebc7d406c952e5ee8295bfa52"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "5a46cf996036a979802b363263021244"
server_key_iv: "f1c9d1799a8bee0cdeabb66ec7184a43"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.448328 16399 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:48:43.450704 16414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.451942 16399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:48:43.452071 16399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "d9c8f08ebc7d406c952e5ee8295bfa52"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "5a46cf996036a979802b363263021244"
server_key_iv: "f1c9d1799a8bee0cdeabb66ec7184a43"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.452180 16399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:43.464129 16399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:43.466665 16399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:43.466872 16399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:43.467418 16399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:43.468232 16399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:43.468299 16399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.468360 16399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:43.468400 16399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.477018 16399 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:46715
I20260925 22:48:43.477047 16527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:46715 every 8 connection(s)
I20260925 22:48:43.477824 16399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:43.481487 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 16399
I20260925 22:48:43.481606 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:48:43.481946 806 external_mini_cluster.cc:1428] Setting key 706ce5b34a1c8353aa011c184928386e
Sep 25 22:48:43 dist-test-slave-vdtf krb5kdc[16158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
I20260925 22:48:43.492560 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.479520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:54245 (local address 127.0.201.190:37517)
0925 22:48:43.479640 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.479644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.480843 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 22:48:43.482011 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.482016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.482085 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.482153 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:43.483597 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.484157 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.485638 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.485956 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.486068 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.488835 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:43.488838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:43.488840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:43.488863 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:43.490302 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.490893 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.490895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.490896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.490976 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.491359 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.491361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.491362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.491539 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:43.491613 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.492280 (+ 667us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.492380 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:48:43.493456 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.479857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:54245)
0925 22:48:43.480699 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.480741 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.481743 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.482356 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.482364 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.482826 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:43.483439 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.483460 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.484278 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.484284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.485504 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.485514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.486105 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.486109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.486284 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.486889 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:43.486904 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:43.488676 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:43.490417 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.490422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.490437 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.490794 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.491065 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.491070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.491072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.491252 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.491647 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:43.491656 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:43.492024 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.492241 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.492615 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":521,"threads_started":1}
I20260925 22:48:43.494616 16528 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37517
I20260925 22:48:43.494845 16528 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
I20260925 22:48:43.495393 16528 heartbeater.cc:511] Master 127.0.201.190:37517 requested a full tablet report, sending...
I20260925 22:48:43.496475 16203 ts_manager.cc:194] Registered new tserver with Master: d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715)
I20260925 22:48:43.497042 16203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:54245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:48:43.511231 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37517
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:43.622705 16533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:43.623049 16533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:43.623124 16533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:43.627326 16533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:43.627424 16533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:43.627519 16533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:48:43.633038 16533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:48:43.634084 16533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:43.635085 16533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:43.644613 16538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:43.644647 16541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:43.644625 16539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.644857 16533 server_base.cc:1034] running on GCE node
I20260925 22:48:43.645280 16533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:43.645879 16533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:43.647059 16533 hybrid_clock.cc:648] HybridClock initialized: now 1790376523647032 us; error 38 us; skew 500 ppm
Sep 25 22:48:43 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:43.649688 16533 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:48:43.650610 16533 webserver.cc:460] Webserver started at http://127.0.201.131:33119/ using document root <none> and password file <none>
I20260925 22:48:43.651098 16533 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:43.651153 16533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:43.651309 16533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:43.653247 16533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "847b346fc4a11e19e01cf29dd4c12598"
server_key_iv: "efe1e218240f4686ab51ca00dfc8d7a1"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.653728 16533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "847b346fc4a11e19e01cf29dd4c12598"
server_key_iv: "efe1e218240f4686ab51ca00dfc8d7a1"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.657302 16533 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:48:43.659471 16548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.660655 16533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:48:43.660750 16533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "847b346fc4a11e19e01cf29dd4c12598"
server_key_iv: "efe1e218240f4686ab51ca00dfc8d7a1"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.660830 16533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:43.682730 16533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:43.685518 16533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:43.685662 16533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:43.686172 16533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:43.686905 16533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:43.686949 16533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.686987 16533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:43.687004 16533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.695386 16533 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:44729
I20260925 22:48:43.695423 16661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:44729 every 8 connection(s)
I20260925 22:48:43.696224 16533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:43.697027 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 16533
I20260925 22:48:43.697147 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:48:43.697422 806 external_mini_cluster.cc:1428] Setting key ae511e45ee8b3433ca36d8b7feeb0fb2
Sep 25 22:48:43 dist-test-slave-vdtf krb5kdc[16158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:48:43.709805 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.698919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:60169 (local address 127.0.201.190:37517)
0925 22:48:43.699039 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.699044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.700168 (+ 1124us) server_negotiation.cc:407] Connection header received
0925 22:48:43.701133 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.701136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.701191 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.701271 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:43.702368 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.702900 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.704067 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.704290 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.704397 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.706777 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:43.706780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:43.706782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:43.706806 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:43.708105 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.708550 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.708552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.708554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.708628 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.708904 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.708906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.708907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.709065 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:43.709138 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.709549 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.709649 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:48:43.710623 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.699305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:60169)
0925 22:48:43.700037 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.700078 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.700943 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.701434 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.701441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.701777 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:43.702233 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.702245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.703021 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.703024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.703946 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.703955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.704431 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.704435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.704600 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.705173 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:43.705189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:43.706628 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:43.708202 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.708207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.708217 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.708466 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.708701 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.708703 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.708710 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.708831 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.709145 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:43.709150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:43.709359 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.709508 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.709733 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":146,"threads_started":1}
I20260925 22:48:43.711656 16662 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37517
I20260925 22:48:43.711897 16662 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:43.712311 16662 heartbeater.cc:511] Master 127.0.201.190:37517 requested a full tablet report, sending...
I20260925 22:48:43.713202 16203 ts_manager.cc:194] Registered new tserver with Master: d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729)
I20260925 22:48:43.713713 16203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:60169
I20260925 22:48:43.722755 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:43.731488 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.724795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.0.201.190:37517)
0925 22:48:43.724933 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.724940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.725079 (+ 139us) server_negotiation.cc:407] Connection header received
0925 22:48:43.725174 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.725176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.725212 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.725275 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:43.725905 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.726381 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.727129 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.727326 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.727395 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.728621 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:43.728624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:43.728626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:43.728647 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:43.730002 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.730426 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.730428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.730430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.730496 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.730725 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.730727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.730728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.730941 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:43.731039 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.731218 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.731306 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:48:43.734227 16203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:48:43.737224 16203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:48:43.751411 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.745723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:44729 (local address 127.0.0.1:56716)
0925 22:48:43.746714 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.746726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.746803 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.747416 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.747420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.747447 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:43.747557 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.747563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.748583 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.748589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.749904 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.749914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.750848 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.750854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.750981 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.751024 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.751135 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.751255 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260925 22:48:43.751422 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.746206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.201.131:44729)
0925 22:48:43.746507 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.746519 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.746888 (+ 369us) server_negotiation.cc:407] Connection header received
0925 22:48:43.746983 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.746990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.747164 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.747303 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:43.747723 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.748454 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.750023 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.750729 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.750841 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.750994 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.751176 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.751269 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":129,"threads_started":1}
I20260925 22:48:43.752188 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.745503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.0.201.130:46715)
0925 22:48:43.745932 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.745945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.746473 (+ 528us) server_negotiation.cc:407] Connection header received
0925 22:48:43.746678 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.746684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.746908 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.747038 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:43.747641 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.748866 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.750729 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.751537 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.751636 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.751783 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.751939 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.752045 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":130,"threads_started":1}
I20260925 22:48:43.752534 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.746115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.0.201.129:37303)
0925 22:48:43.746741 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.746756 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.746779 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:48:43.746855 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.746860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.747069 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.747238 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:43.747637 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.748866 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.750908 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.751767 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.751864 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.752002 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.752192 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.752373 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":68,"threads_started":1}
I20260925 22:48:43.752547 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.745364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:46715 (local address 127.0.0.1:34898)
0925 22:48:43.746317 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.746370 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.746532 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.747248 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.747253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.747284 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:43.747500 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.747508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.749475 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.749478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.750610 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.750619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.751647 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.751651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.751761 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.751783 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.752329 (+ 546us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.752408 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":66,"spinlock_wait_cycles":3968,"thread_start_us":74,"threads_started":1}
I20260925 22:48:43.752554 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.745456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:37303 (local address 127.0.0.1:40386)
0925 22:48:43.746317 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.746370 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.746502 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.747242 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.747248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.747280 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:43.747500 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.747508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.749008 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.749011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.750769 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.750775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.751951 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.751955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.752053 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.752077 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.752164 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.752405 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":20,"spinlock_wait_cycles":18560,"thread_start_us":106,"threads_started":1}
I20260925 22:48:43.754875 16596 tablet_service.cc:1467] Processing CreateTablet for tablet e36b4e299e024751a29568920929e27c (DEFAULT_TABLE table=test-table [id=7bf6abedecec47b09ac756048f4ca116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:43.755239 16462 tablet_service.cc:1467] Processing CreateTablet for tablet e36b4e299e024751a29568920929e27c (DEFAULT_TABLE table=test-table [id=7bf6abedecec47b09ac756048f4ca116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:43.755973 16596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e36b4e299e024751a29568920929e27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:43.756297 16462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e36b4e299e024751a29568920929e27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:43.756016 16328 tablet_service.cc:1467] Processing CreateTablet for tablet e36b4e299e024751a29568920929e27c (DEFAULT_TABLE table=test-table [id=7bf6abedecec47b09ac756048f4ca116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:43.756943 16328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e36b4e299e024751a29568920929e27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:43.761868 16679 tablet_bootstrap.cc:492] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: Bootstrap starting.
I20260925 22:48:43.763513 16681 tablet_bootstrap.cc:492] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Bootstrap starting.
I20260925 22:48:43.764299 16680 tablet_bootstrap.cc:492] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: Bootstrap starting.
I20260925 22:48:43.764747 16679 tablet_bootstrap.cc:654] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:43.765596 16679 log.cc:824] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:43.765898 16681 tablet_bootstrap.cc:654] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:43.766526 16680 tablet_bootstrap.cc:654] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:43.767025 16681 log.cc:824] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:43.767287 16680 log.cc:824] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:43.767803 16679 tablet_bootstrap.cc:492] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: No bootstrap required, opened a new log
I20260925 22:48:43.768010 16679 ts_tablet_manager.cc:1397] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:48:43.768628 16681 tablet_bootstrap.cc:492] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: No bootstrap required, opened a new log
I20260925 22:48:43.768747 16681 ts_tablet_manager.cc:1397] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:48:43.769155 16680 tablet_bootstrap.cc:492] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: No bootstrap required, opened a new log
I20260925 22:48:43.769307 16680 ts_tablet_manager.cc:1397] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:48:43.771755 16681 raft_consensus.cc:348] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
I20260925 22:48:43.771979 16681 raft_consensus.cc:374] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:43.772051 16681 raft_consensus.cc:729] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028d8019ade7490c8229a79f84f0bf29, State: Initialized, Role: FOLLOWER
I20260925 22:48:43.771906 16679 raft_consensus.cc:348] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
I20260925 22:48:43.772215 16679 raft_consensus.cc:374] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:43.772311 16679 raft_consensus.cc:729] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d92a34f6f7402aa2a9d198faaa7d3e, State: Initialized, Role: FOLLOWER
I20260925 22:48:43.772493 16681 consensus_queue.cc:260] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
I20260925 22:48:43.772785 16679 consensus_queue.cc:260] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
I20260925 22:48:43.773140 16394 heartbeater.cc:503] Master 127.0.201.190:37517 was elected leader, sending a full tablet report...
I20260925 22:48:43.773031 16680 raft_consensus.cc:348] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
I20260925 22:48:43.773263 16680 raft_consensus.cc:374] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:43.773312 16680 raft_consensus.cc:729] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c8f08ebc7d406c952e5ee8295bfa52, State: Initialized, Role: FOLLOWER
I20260925 22:48:43.773375 16681 ts_tablet_manager.cc:1428] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:48:43.773715 16680 consensus_queue.cc:260] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
W20260925 22:48:43.773918 16395 tablet.cc:2369] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:43.774315 16528 heartbeater.cc:503] Master 127.0.201.190:37517 was elected leader, sending a full tablet report...
I20260925 22:48:43.775177 16679 ts_tablet_manager.cc:1428] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:48:43.775775 16662 heartbeater.cc:503] Master 127.0.201.190:37517 was elected leader, sending a full tablet report...
I20260925 22:48:43.776297 16680 ts_tablet_manager.cc:1428] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:48:43.899399 16687 raft_consensus.cc:482] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:43.899653 16687 raft_consensus.cc:504] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
I20260925 22:48:43.901229 16687 leader_election.cc:288] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715), 028d8019ade7490c8229a79f84f0bf29 (127.0.201.129:37303)
I20260925 22:48:43.904765 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.901109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:37303 (local address 127.0.201.131:51417)
0925 22:48:43.901221 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.901237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.901328 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.901605 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.901608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.901626 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:43.901761 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.901766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.902699 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.902702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.903594 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.903608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.904351 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.904355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.904461 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.904486 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.904563 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.904632 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":3}
I20260925 22:48:43.904814 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.901211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:51417 (local address 127.0.201.129:37303)
0925 22:48:43.901345 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.901349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.901398 (+ 49us) server_negotiation.cc:407] Connection header received
0925 22:48:43.901461 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.901464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.901507 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.901595 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:43.901922 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.902595 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.903737 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.904236 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.904327 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.904367 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.904589 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.904675 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:48:43.905022 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.901109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:46715 (local address 127.0.201.131:55421)
0925 22:48:43.901730 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.901742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.901823 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.902057 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.902060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.902078 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:43.902191 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.902197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.902977 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.902980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.903874 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.903884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.904604 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.904607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.904702 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.904725 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.904801 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.904874 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":305,"thread_start_us":130,"threads_started":1}
I20260925 22:48:43.905061 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.901221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:55421 (local address 127.0.201.130:46715)
0925 22:48:43.901356 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.901361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.901832 (+ 471us) server_negotiation.cc:407] Connection header received
0925 22:48:43.901921 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.901923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.901967 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.902027 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:43.902297 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.902865 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.904011 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.904481 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.904597 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.904639 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.904833 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.904919 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:48:43.905633 16348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36b4e299e024751a29568920929e27c" candidate_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "028d8019ade7490c8229a79f84f0bf29" is_pre_election: true
I20260925 22:48:43.905793 16482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36b4e299e024751a29568920929e27c" candidate_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" is_pre_election: true
I20260925 22:48:43.905956 16348 raft_consensus.cc:2454] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d92a34f6f7402aa2a9d198faaa7d3e in term 0.
I20260925 22:48:43.906090 16482 raft_consensus.cc:2454] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d92a34f6f7402aa2a9d198faaa7d3e in term 0.
I20260925 22:48:43.906474 16551 leader_election.cc:302] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 028d8019ade7490c8229a79f84f0bf29, d8d92a34f6f7402aa2a9d198faaa7d3e, d9c8f08ebc7d406c952e5ee8295bfa52; no voters:
I20260925 22:48:43.906744 16687 raft_consensus.cc:2781] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:43.906819 16687 raft_consensus.cc:482] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:43.906883 16687 raft_consensus.cc:3037] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:43.907963 16687 raft_consensus.cc:504] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
I20260925 22:48:43.908324 16687 leader_election.cc:288] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [CANDIDATE]: Term 1 election: Requested vote from peers d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715), 028d8019ade7490c8229a79f84f0bf29 (127.0.201.129:37303)
I20260925 22:48:43.908633 16482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36b4e299e024751a29568920929e27c" candidate_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52"
I20260925 22:48:43.908720 16348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36b4e299e024751a29568920929e27c" candidate_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "028d8019ade7490c8229a79f84f0bf29"
I20260925 22:48:43.908733 16482 raft_consensus.cc:3037] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:43.908851 16348 raft_consensus.cc:3037] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:43.909794 16482 raft_consensus.cc:2454] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d92a34f6f7402aa2a9d198faaa7d3e in term 1.
I20260925 22:48:43.909806 16348 raft_consensus.cc:2454] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d92a34f6f7402aa2a9d198faaa7d3e in term 1.
I20260925 22:48:43.910095 16551 leader_election.cc:302] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 028d8019ade7490c8229a79f84f0bf29, d8d92a34f6f7402aa2a9d198faaa7d3e; no voters:
I20260925 22:48:43.910286 16687 raft_consensus.cc:2781] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:43.910507 16687 raft_consensus.cc:686] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 1 LEADER]: Becoming Leader. State: Replica: d8d92a34f6f7402aa2a9d198faaa7d3e, State: Running, Role: LEADER
I20260925 22:48:43.910857 16687 consensus_queue.cc:237] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
I20260925 22:48:43.913854 16203 catalog_manager.cc:5495] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131). New cstate: current_term: 1 leader_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } health_report { overall_health: HEALTHY } } }
I20260925 22:48:43.942554 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.939357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.0.201.131:44729)
0925 22:48:43.939472 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.939476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.939490 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:48:43.939563 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.939565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.939616 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.939764 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:43.939970 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.940461 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.941244 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.941450 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.941522 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.941768 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:43.942173 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:43.942260 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.942307 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.942353 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 22:48:43.947048 16663 tablet.cc:2369] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:43.948247 16348 raft_consensus.cc:1261] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Refusing update from remote peer d8d92a34f6f7402aa2a9d198faaa7d3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:43.948246 16482 raft_consensus.cc:1261] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52 [term 1 FOLLOWER]: Refusing update from remote peer d8d92a34f6f7402aa2a9d198faaa7d3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:43.948974 16689 consensus_queue.cc:1035] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:43.949142 16687 consensus_queue.cc:1035] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:43.957914 16693 mvcc.cc:205] Tried to move back new op lower bound from 7333382242084790272 to 7333382241941315584. Current Snapshot: MvccSnapshot[applied={T|T < 7333382242084790272}]
I20260925 22:48:43.966202 16201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:43.966372 16201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:43.968779 16201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Sending DeleteTablet for 3 replicas of tablet e36b4e299e024751a29568920929e27c
I20260925 22:48:43.969529 16328 tablet_service.cc:1514] Processing DeleteTablet for tablet e36b4e299e024751a29568920929e27c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:43 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:40386
I20260925 22:48:43.969528 16462 tablet_service.cc:1514] Processing DeleteTablet for tablet e36b4e299e024751a29568920929e27c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:43 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:34898
I20260925 22:48:43.970027 16702 tablet_replica.cc:330] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: stopping tablet replica
I20260925 22:48:43.970155 16596 tablet_service.cc:1514] Processing DeleteTablet for tablet e36b4e299e024751a29568920929e27c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:43 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:56716
I20260925 22:48:43.970324 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 16265
I20260925 22:48:43.970429 16702 raft_consensus.cc:2229] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:43.971266 16703 tablet_replica.cc:330] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: stopping tablet replica
I20260925 22:48:43.971552 16703 raft_consensus.cc:2229] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:43.971585 16704 tablet_replica.cc:330] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: stopping tablet replica
I20260925 22:48:43.971884 16704 raft_consensus.cc:2229] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:48:43.971987 16703 raft_consensus.cc:2258] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:43.972345 16704 raft_consensus.cc:2258] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:43.973645 16703 ts_tablet_manager.cc:1905] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:43.973645 16704 ts_tablet_manager.cc:1905] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:43.976459 16704 ts_tablet_manager.cc:1918] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:43.976564 16704 log.cc:1196] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/e36b4e299e024751a29568920929e27c
I20260925 22:48:43.976550 16703 ts_tablet_manager.cc:1918] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:43.976632 16703 log.cc:1196] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/e36b4e299e024751a29568920929e27c
W20260925 22:48:43.976699 16189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:37303 (error 108)
I20260925 22:48:43.976970 16704 ts_tablet_manager.cc:1939] T e36b4e299e024751a29568920929e27c P d8d92a34f6f7402aa2a9d198faaa7d3e: Deleting consensus metadata
I20260925 22:48:43.976970 16703 ts_tablet_manager.cc:1939] T e36b4e299e024751a29568920929e27c P d9c8f08ebc7d406c952e5ee8295bfa52: Deleting consensus metadata
I20260925 22:48:43.977690 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.977392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:37303 (local address 127.0.0.1:40390)
0925 22:48:43.977515 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.977589 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:48:43.977845 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 16399
W20260925 22:48:43.977942 16189 catalog_manager.cc:4584] TS 028d8019ade7490c8229a79f84f0bf29 (127.0.201.129:37303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e36b4e299e024751a29568920929e27c: Network error: Client connection negotiation failed: client connection to 127.0.201.129:37303: connect: Connection refused (error 111)
I20260925 22:48:43.978173 16190 catalog_manager.cc:4841] TS d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729): tablet e36b4e299e024751a29568920929e27c (table test-table [id=7bf6abedecec47b09ac756048f4ca116]) successfully deleted
I20260925 22:48:43.985428 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 16533
I20260925 22:48:43.986299 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.985636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:46715 (local address 127.0.0.1:34910)
0925 22:48:43.986124 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.986187 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.130:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389}
I20260925 22:48:43.991412 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 16171
I20260925 22:49:03.997706 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37517
--webserver_interface=127.0.201.190
--webserver_port=46511
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:37517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:04.110272 16706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:04.110546 16706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:04.110586 16706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:04.114711 16706 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:49:04.114770 16706 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:49:04.114787 16706 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:49:04.114801 16706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:04.114818 16706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:04.114837 16706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:04.114897 16706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:04.114918 16706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:04.120769 16706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:37517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:04.121862 16706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:04.122916 16706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:04.136639 16712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:04.136719 16706 server_base.cc:1034] running on GCE node
W20260925 22:49:04.136646 16715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:04.136639 16713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:04.137704 16706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:04.138950 16706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:04.140131 16706 hybrid_clock.cc:648] HybridClock initialized: now 1790376544140104 us; error 53 us; skew 500 ppm
Sep 25 22:49:04 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:04.143066 16706 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:49:04.144129 16706 webserver.cc:460] Webserver started at http://127.0.201.190:46511/ using document root <none> and password file <none>
I20260925 22:49:04.144724 16706 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:04.144796 16706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:04.148725 16706 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:04.150870 16722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:04.152148 16706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:04.152323 16706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "1facd2cfc739458db9d7a62a276c0fef"
format_stamp: "Formatted at 2026-09-25 22:48:42 on dist-test-slave-vdtf"
server_key: "6bf12e35920326c8c0abf84718575c46"
server_key_iv: "1c324b65749d926b7008eeb612f38b4e"
server_key_version: "encryptionkey@0"
I20260925 22:49:04.152925 16706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:04.168565 16706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:04.171393 16706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:04.171589 16706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:04.178767 16706 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:37517
I20260925 22:49:04.179080 16774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:37517 every 8 connection(s)
I20260925 22:49:04.179865 16706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:04.184144 16775 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:49:04.184505 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 16706
I20260925 22:49:04.184949 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:37303
--local_ip_for_outbound_sockets=127.0.201.129
--tserver_master_addrs=127.0.201.190:37517
--webserver_port=36003
--webserver_interface=127.0.201.129
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:04.185952 16775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Bootstrap starting.
I20260925 22:49:04.202967 16775 log.cc:824] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:04.211522 16775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:49:04.211956 16775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Bootstrap complete.
I20260925 22:49:04.215955 16775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:49:04.216377 16775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1facd2cfc739458db9d7a62a276c0fef, State: Initialized, Role: FOLLOWER
I20260925 22:49:04.216915 16775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:49:04.217069 16775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:04.217164 16775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:04.217286 16775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:49:04.219163 16775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:49:04.219514 16775 leader_election.cc:302] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1facd2cfc739458db9d7a62a276c0fef; no voters:
I20260925 22:49:04.219883 16775 leader_election.cc:288] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:49:04.219967 16779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:49:04.220191 16779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 2 LEADER]: Becoming Leader. State: Replica: 1facd2cfc739458db9d7a62a276c0fef, State: Running, Role: LEADER
I20260925 22:49:04.220530 16779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:49:04.221068 16775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:04.221832 16780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1facd2cfc739458db9d7a62a276c0fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } } }
I20260925 22:49:04.221951 16780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:04.222314 16781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1facd2cfc739458db9d7a62a276c0fef. Latest consensus state: current_term: 2 leader_uuid: "1facd2cfc739458db9d7a62a276c0fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } } }
I20260925 22:49:04.222445 16781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:04.222931 16788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:04.226909 16788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:04.227419 16788 catalog_manager.cc:1259] Loaded cluster ID: 8e98aab1064d463696c76819421d324d
I20260925 22:49:04.227500 16788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:04.229389 16788 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:04.229978 16788 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Loaded TSK: 0
I20260925 22:49:04.239527 16788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Generated new TSK 1
I20260925 22:49:04.239813 16788 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:49:04.309116 16777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:04.309384 16777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:04.309427 16777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:04.313484 16777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:04.313545 16777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:04.313627 16777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:49:04.318789 16777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:37303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=36003
--tserver_master_addrs=127.0.201.190:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:04.319888 16777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:04.320993 16777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:04.331653 16802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:04.331773 16803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:04.331661 16805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:04.331730 16777 server_base.cc:1034] running on GCE node
I20260925 22:49:04.333266 16777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:04.333896 16777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:04.335086 16777 hybrid_clock.cc:648] HybridClock initialized: now 1790376544335064 us; error 40 us; skew 500 ppm
Sep 25 22:49:04 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:04.337924 16777 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:49:04.338847 16777 webserver.cc:460] Webserver started at http://127.0.201.129:36003/ using document root <none> and password file <none>
I20260925 22:49:04.339406 16777 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:04.339516 16777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:04.343415 16777 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:04.345678 16812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:04.346959 16777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 22:49:04.347105 16777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "028d8019ade7490c8229a79f84f0bf29"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "bb603a8e20f77fea6234ca7a16931224"
server_key_iv: "8270cc92bf22e150afd22fa0fd83225b"
server_key_version: "encryptionkey@0"
I20260925 22:49:04.347632 16777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:04.360044 16777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:04.363277 16777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:04.363483 16777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:04.364137 16777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:04.365432 16819 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:49:04.369480 16777 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:49:04.369542 16777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260925 22:49:04.369614 16777 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:49:04.371228 16777 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:49:04.371282 16777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:49:04.371524 16819 tablet_bootstrap.cc:492] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Bootstrap starting.
I20260925 22:49:04.382035 16777 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:37303
I20260925 22:49:04.382169 16926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:37303 every 8 connection(s)
I20260925 22:49:04.383210 16777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:04.391530 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 16777
I20260925 22:49:04.391806 16819 log.cc:824] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:04.392071 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:46715
--local_ip_for_outbound_sockets=127.0.201.130
--tserver_master_addrs=127.0.201.190:37517
--webserver_port=45939
--webserver_interface=127.0.201.130
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:49:04 dist-test-slave-vdtf krb5kdc[16158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:04.400768 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.385545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:34721)
0925 22:49:04.386200 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:04.386258 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:04.387108 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:04.388463 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:04.388471 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:04.388934 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:04.389393 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.389407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.390706 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.390709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.391902 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.391913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.393005 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.393009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.393197 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.393993 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:04.394013 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:04.395591 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:04.397752 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:04.397758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:04.397771 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:04.398024 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:04.398389 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:04.398392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:04.398394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:04.398527 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:04.399081 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:04.399087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:04.399324 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 22:49:04.399499 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 22:49:04.399843 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":151,"threads_started":1}
I20260925 22:49:04.400753 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.385557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:34721 (local address 127.0.201.190:37517)
0925 22:49:04.385969 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.385982 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.386391 (+ 409us) server_negotiation.cc:407] Connection header received
0925 22:49:04.387526 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.387537 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.387933 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.388415 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:04.389580 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.390534 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.392049 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.392841 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.393034 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.395865 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:04.395871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:04.395883 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:04.395911 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:04.397609 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.398146 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.398152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.398159 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.398283 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.398623 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.398627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.398629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.398964 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:04.399109 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.399621 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.399909 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":95,"threads_started":1}
I20260925 22:49:04.401448 16819 tablet_bootstrap.cc:492] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:49:04.401916 16819 tablet_bootstrap.cc:492] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Bootstrap complete.
I20260925 22:49:04.403177 16927 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37517
I20260925 22:49:04.403396 16927 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:04.403402 16819 ts_tablet_manager.cc:1397] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260925 22:49:04.403959 16927 heartbeater.cc:511] Master 127.0.201.190:37517 requested a full tablet report, sending...
I20260925 22:49:04.406211 16739 ts_manager.cc:194] Registered new tserver with Master: 028d8019ade7490c8229a79f84f0bf29 (127.0.201.129:37303)
I20260925 22:49:04.406447 16819 raft_consensus.cc:348] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
I20260925 22:49:04.406967 16819 raft_consensus.cc:729] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028d8019ade7490c8229a79f84f0bf29, State: Initialized, Role: FOLLOWER
I20260925 22:49:04.407441 16819 consensus_queue.cc:260] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } } peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } }
I20260925 22:49:04.408437 16819 ts_tablet_manager.cc:1428] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260925 22:49:04.409840 16739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:34721
I20260925 22:49:04.412106 16927 heartbeater.cc:503] Master 127.0.201.190:37517 was elected leader, sending a full tablet report...
Sep 25 22:49:04 dist-test-slave-vdtf krb5kdc[16158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.129@KRBTEST.COM
I20260925 22:49:04.420233 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.409282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.0.201.129:37303)
0925 22:49:04.410166 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.410176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.410195 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:49:04.410254 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.410258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.410453 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.410572 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:04.411518 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.412727 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.413854 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.414371 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.414477 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.416854 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:04.416862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:04.416872 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:04.416893 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:04.418377 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.418893 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.418897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.418899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.418970 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.419246 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.419248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.419249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.419465 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:04.419562 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.419973 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.420102 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":81,"threads_started":1}
I20260925 22:49:04.420271 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.409187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:37303 (local address 127.0.0.1:38264)
0925 22:49:04.409745 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:04.409795 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:04.409919 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:04.410721 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:04.410726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:04.411158 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:04.411363 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.411376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.412841 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.412844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.413749 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.413758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.414478 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.414481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.414556 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.415122 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:04.415133 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:04.416722 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:04.418484 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:04.418496 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:04.418503 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:04.418787 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:04.419054 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:04.419057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:04.419059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:04.419151 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:04.419570 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:04.419576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:04.419761 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 22:49:04.419914 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:49:04.420107 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":90,"threads_started":1}
I20260925 22:49:04.421452 16861 tablet_service.cc:1514] Processing DeleteTablet for tablet e36b4e299e024751a29568920929e27c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:43 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:38264
I20260925 22:49:04.422348 16936 tablet_replica.cc:330] stopping tablet replica
I20260925 22:49:04.422643 16936 raft_consensus.cc:2229] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:04.422904 16936 raft_consensus.cc:2258] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:04.423938 16936 ts_tablet_manager.cc:1905] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:04.427526 16936 ts_tablet_manager.cc:1918] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:04.427640 16936 log.cc:1196] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal/wals/e36b4e299e024751a29568920929e27c
I20260925 22:49:04.427918 16936 ts_tablet_manager.cc:1939] T e36b4e299e024751a29568920929e27c P 028d8019ade7490c8229a79f84f0bf29: Deleting consensus metadata
I20260925 22:49:04.429735 16725 catalog_manager.cc:4841] TS 028d8019ade7490c8229a79f84f0bf29 (127.0.201.129:37303): tablet e36b4e299e024751a29568920929e27c (table test-table [id=7bf6abedecec47b09ac756048f4ca116]) successfully deleted
W20260925 22:49:04.509632 16932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:04.509903 16932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:04.509945 16932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:04.513888 16932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:04.513938 16932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:04.513994 16932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:49:04.518937 16932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:46715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=45939
--tserver_master_addrs=127.0.201.190:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:04.519939 16932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:04.520881 16932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:04.530964 16944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:04.531009 16941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:04.530999 16942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:04.531261 16932 server_base.cc:1034] running on GCE node
I20260925 22:49:04.531637 16932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:04.532280 16932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:04.533471 16932 hybrid_clock.cc:648] HybridClock initialized: now 1790376544533438 us; error 46 us; skew 500 ppm
Sep 25 22:49:04 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:04.536262 16932 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:49:04.537128 16932 webserver.cc:460] Webserver started at http://127.0.201.130:45939/ using document root <none> and password file <none>
I20260925 22:49:04.537635 16932 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:04.537703 16932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:04.541450 16932 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:04.543423 16951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:04.544884 16932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:04.544986 16932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "d9c8f08ebc7d406c952e5ee8295bfa52"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "5a46cf996036a979802b363263021244"
server_key_iv: "f1c9d1799a8bee0cdeabb66ec7184a43"
server_key_version: "encryptionkey@0"
I20260925 22:49:04.545559 16932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:04.571092 16932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:04.574335 16932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:04.574554 16932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:04.575181 16932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:04.576023 16932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:04.576097 16932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:04.576174 16932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:04.576222 16932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:04.585505 16932 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:46715
I20260925 22:49:04.585556 17064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:46715 every 8 connection(s)
I20260925 22:49:04.586498 16932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:04.589334 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 16932
I20260925 22:49:04.589864 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:44729
--local_ip_for_outbound_sockets=127.0.201.131
--tserver_master_addrs=127.0.201.190:37517
--webserver_port=33119
--webserver_interface=127.0.201.131
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:49:04 dist-test-slave-vdtf krb5kdc[16158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:04.602867 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.588297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:59091 (local address 127.0.201.190:37517)
0925 22:49:04.588435 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.588439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.589558 (+ 1119us) server_negotiation.cc:407] Connection header received
0925 22:49:04.590898 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.590901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.590954 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.591034 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:04.592359 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.593245 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.594375 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.594841 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.594968 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.597874 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:04.597881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:04.597886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:04.597940 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:04.600781 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.601309 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.601316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.601320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.601416 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.601738 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.601741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.601743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.601981 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:04.602075 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.602551 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.602688 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:04.603721 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.588626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:59091)
0925 22:49:04.589363 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:04.589425 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:04.590641 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:04.591268 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:04.591280 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:04.591727 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:04.592203 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.592223 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.593380 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.593383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.594245 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.594254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.595000 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.595003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.595143 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.595816 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:04.595833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:04.597612 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:04.600923 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:04.600929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:04.600942 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:04.601204 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:04.601508 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:04.601510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:04.601512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:04.601645 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:04.602076 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:04.602081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:04.602329 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 22:49:04.602487 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 22:49:04.602746 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":100,"threads_started":1}
I20260925 22:49:04.604905 17065 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37517
I20260925 22:49:04.605160 17065 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:04.605654 17065 heartbeater.cc:511] Master 127.0.201.190:37517 requested a full tablet report, sending...
I20260925 22:49:04.606655 16739 ts_manager.cc:194] Registered new tserver with Master: d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715)
I20260925 22:49:04.607223 16739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:59091
W20260925 22:49:04.704288 17068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:04.704572 17068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:04.704643 17068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:04.708673 17068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:04.708755 17068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:04.708842 17068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:49:04.713927 17068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:44729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=33119
--tserver_master_addrs=127.0.201.190:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:04.714985 17068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:04.716084 17068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:04.726033 17076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:04.726044 17073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:04.726034 17074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:04.726395 17068 server_base.cc:1034] running on GCE node
I20260925 22:49:04.726809 17068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:04.727526 17068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:04.728688 17068 hybrid_clock.cc:648] HybridClock initialized: now 1790376544728661 us; error 47 us; skew 500 ppm
Sep 25 22:49:04 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:04.731422 17068 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:49:04.732362 17068 webserver.cc:460] Webserver started at http://127.0.201.131:33119/ using document root <none> and password file <none>
I20260925 22:49:04.732975 17068 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:04.733083 17068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:04.736846 17068 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 22:49:04.739008 17083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:04.740388 17068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:04.740540 17068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "847b346fc4a11e19e01cf29dd4c12598"
server_key_iv: "efe1e218240f4686ab51ca00dfc8d7a1"
server_key_version: "encryptionkey@0"
I20260925 22:49:04.741165 17068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:04.767793 17068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:04.770774 17068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:04.770990 17068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:04.771627 17068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:04.772459 17068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:04.772528 17068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:04.772598 17068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:04.772648 17068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:04.781780 17068 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:44729
I20260925 22:49:04.781803 17196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:44729 every 8 connection(s)
I20260925 22:49:04.783030 17068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:04.787024 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 17068
Sep 25 22:49:04 dist-test-slave-vdtf krb5kdc[16158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:04.796929 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.787766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.0.201.190:37517)
0925 22:49:04.787985 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.787988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.788229 (+ 241us) server_negotiation.cc:407] Connection header received
0925 22:49:04.788347 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.788349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.788384 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.788456 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:04.789343 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.790059 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.791244 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.791552 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.791659 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.793382 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:04.793385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:04.793387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:04.793409 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:04.795186 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.795599 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.795604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.795606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.795742 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.796041 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.796044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.796046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.796234 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:04.796458 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.796634 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.796756 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
I20260925 22:49:04.798375 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.784907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:55783 (local address 127.0.201.190:37517)
0925 22:49:04.785058 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.785062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.786329 (+ 1267us) server_negotiation.cc:407] Connection header received
0925 22:49:04.787293 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.787296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.787356 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.787460 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:04.789110 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.789884 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.791446 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.791818 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.791892 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.794696 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:04.794701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:04.794706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:04.794747 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:04.796159 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.796693 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.796697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.796698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.796778 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.797196 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.797200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.797201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.797419 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:04.797512 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.798122 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.798238 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:49:04.799296 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.785286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:55783)
0925 22:49:04.786166 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:04.786207 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:04.787009 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:04.787666 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:04.787699 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:04.788219 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:04.788950 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.788974 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.790002 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.790005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.791342 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.791351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.791966 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.791970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.792153 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.792632 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:04.792655 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:04.794520 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:04.796281 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:04.796287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:04.796301 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:04.796554 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:04.796906 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:04.796911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:04.796916 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:04.797102 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:04.797529 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:04.797535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:04.797854 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 22:49:04.798081 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 22:49:04.798414 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":546,"threads_started":1}
I20260925 22:49:04.800331 17197 heartbeater.cc:347] Connected to a master server at 127.0.201.190:37517
I20260925 22:49:04.800545 17197 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:04.800973 17197 heartbeater.cc:511] Master 127.0.201.190:37517 requested a full tablet report, sending...
I20260925 22:49:04.801887 16739 ts_manager.cc:194] Registered new tserver with Master: d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729)
I20260925 22:49:04.802392 16739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:55783
I20260925 22:49:04.810395 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:49:04.814366 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.811020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.0.201.190:37517)
0925 22:49:04.811180 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.811184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.811281 (+ 97us) server_negotiation.cc:407] Connection header received
0925 22:49:04.811428 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.811430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.811525 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.811604 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:04.811961 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.812447 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.813317 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.813519 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.813593 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.813906 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:04.814042 (+ 136us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 22:49:04.814237 (+ 195us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33618: token expired
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 22:49:04.814438 16929 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33618: token expired
W20260925 22:49:04.814815 16666 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.201.190:37517)
I20260925 22:49:04.822124 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.815148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.0.201.190:37517)
0925 22:49:04.815274 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.815280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.815295 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:49:04.815422 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.815425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.815473 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.815556 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:04.816231 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.816718 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.817516 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.817726 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.817833 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.819125 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:04.819129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:04.819131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:04.819172 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:04.820458 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.820902 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.820905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.820907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.820990 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.821275 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.821279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.821281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.821453 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:04.821593 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.821823 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.821958 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:49:04.824961 16739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:04.828233 16739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:04.835875 16861 tablet_service.cc:1467] Processing CreateTablet for tablet 6092341b58004de99f5de4cfe5a29586 (DEFAULT_TABLE table=test-table [id=2ab22f97127d44be90c1728e33e2d684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:04.836537 16861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6092341b58004de99f5de4cfe5a29586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:04.839325 16819 tablet_bootstrap.cc:492] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Bootstrap starting.
I20260925 22:49:04.840242 16819 tablet_bootstrap.cc:654] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:04.840647 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.834480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:44729 (local address 127.0.0.1:37256)
0925 22:49:04.834575 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:04.834592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:04.834669 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:04.836108 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:04.836112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:04.836140 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:04.836318 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.836326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.837825 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.837828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.839118 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.839129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.840159 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.840162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.840289 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.840332 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 22:49:04.840399 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 22:49:04.840489 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":120,"spinlock_wait_cycles":5248}
I20260925 22:49:04.840787 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.834944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:46715 (local address 127.0.0.1:52762)
0925 22:49:04.835161 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:04.835174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:04.835267 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:04.836108 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:04.836112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:04.836141 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:04.836299 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.836307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.837940 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.837943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.838981 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.838991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.840158 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.840162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.840520 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.840541 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:49:04.840605 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:49:04.840664 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":261,"spinlock_wait_cycles":13824,"thread_start_us":82,"threads_started":1}
I20260925 22:49:04.841288 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.835118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.0.201.130:46715)
0925 22:49:04.835559 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.835573 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.835599 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:49:04.835710 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.835714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.835935 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.836449 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:04.836629 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.837833 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.839107 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.840009 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.840240 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.840405 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.841018 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.841126 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":128,"threads_started":1}
I20260925 22:49:04.841353 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.834715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.0.201.131:44729)
0925 22:49:04.835579 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.835591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.835609 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:49:04.835705 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.835711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.835936 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.836219 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:04.836408 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.837706 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.839263 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.840044 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.840795 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.841083 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.841127 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.841206 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":165,"threads_started":1}
I20260925 22:49:04.842875 16819 tablet_bootstrap.cc:492] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: No bootstrap required, opened a new log
I20260925 22:49:04.842971 16819 ts_tablet_manager.cc:1397] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:49:04.843428 16819 raft_consensus.cc:348] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:04.843555 16819 raft_consensus.cc:374] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:04.843595 16819 raft_consensus.cc:729] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028d8019ade7490c8229a79f84f0bf29, State: Initialized, Role: FOLLOWER
I20260925 22:49:04.843742 16819 consensus_queue.cc:260] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:04.844017 16819 ts_tablet_manager.cc:1428] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:49:04.844240 16999 tablet_service.cc:1467] Processing CreateTablet for tablet 6092341b58004de99f5de4cfe5a29586 (DEFAULT_TABLE table=test-table [id=2ab22f97127d44be90c1728e33e2d684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:04.844237 17131 tablet_service.cc:1467] Processing CreateTablet for tablet 6092341b58004de99f5de4cfe5a29586 (DEFAULT_TABLE table=test-table [id=2ab22f97127d44be90c1728e33e2d684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:04.845366 16999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6092341b58004de99f5de4cfe5a29586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:04.845391 17131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6092341b58004de99f5de4cfe5a29586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:04.852272 17210 tablet_bootstrap.cc:492] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: Bootstrap starting.
I20260925 22:49:04.852885 17211 tablet_bootstrap.cc:492] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: Bootstrap starting.
I20260925 22:49:04.854887 17210 tablet_bootstrap.cc:654] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:04.855286 17211 tablet_bootstrap.cc:654] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:04.855651 17210 log.cc:824] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:04.856132 17211 log.cc:824] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:04.857570 17210 tablet_bootstrap.cc:492] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: No bootstrap required, opened a new log
I20260925 22:49:04.857728 17210 ts_tablet_manager.cc:1397] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:49:04.857935 17211 tablet_bootstrap.cc:492] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: No bootstrap required, opened a new log
I20260925 22:49:04.858121 17211 ts_tablet_manager.cc:1397] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:49:04.860746 17210 raft_consensus.cc:348] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:04.860958 17210 raft_consensus.cc:374] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:04.861016 17210 raft_consensus.cc:729] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c8f08ebc7d406c952e5ee8295bfa52, State: Initialized, Role: FOLLOWER
I20260925 22:49:04.861255 17211 raft_consensus.cc:348] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:04.861481 17211 raft_consensus.cc:374] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:04.861538 17211 raft_consensus.cc:729] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d92a34f6f7402aa2a9d198faaa7d3e, State: Initialized, Role: FOLLOWER
I20260925 22:49:04.861466 17210 consensus_queue.cc:260] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:04.861948 17211 consensus_queue.cc:260] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:04.862180 17065 heartbeater.cc:503] Master 127.0.201.190:37517 was elected leader, sending a full tablet report...
I20260925 22:49:04.862573 17210 ts_tablet_manager.cc:1428] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260925 22:49:04.862870 17211 ts_tablet_manager.cc:1428] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:04.863162 17197 heartbeater.cc:503] Master 127.0.201.190:37517 was elected leader, sending a full tablet report...
W20260925 22:49:04.884783 16928 tablet.cc:2369] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:04.932265 17215 raft_consensus.cc:482] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:04.932492 17215 raft_consensus.cc:504] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:04.934438 17215 leader_election.cc:288] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715), 028d8019ade7490c8229a79f84f0bf29 (127.0.201.129:37303)
I20260925 22:49:04.937837 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.934003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:46715 (local address 127.0.201.131:40065)
0925 22:49:04.934201 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:04.934217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:04.934323 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:04.934754 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:04.934757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:04.934783 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:04.934927 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.934933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.935776 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.935779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.936683 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.936692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.937400 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.937404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.937511 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.937536 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:49:04.937630 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:49:04.937700 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":20}
I20260925 22:49:04.938063 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.934007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:37303 (local address 127.0.201.131:51673)
0925 22:49:04.934649 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:04.934664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:04.934756 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:04.935067 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:04.935070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:04.935087 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:04.935194 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.935200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.936012 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.936014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.936913 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:04.936923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.937668 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.937670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:04.937746 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.937764 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:49:04.937813 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 22:49:04.937874 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":316,"thread_start_us":93,"threads_started":1}
I20260925 22:49:04.938086 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.934175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:51673 (local address 127.0.201.129:37303)
0925 22:49:04.934522 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.934526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.934727 (+ 201us) server_negotiation.cc:407] Connection header received
0925 22:49:04.934863 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.934866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.934933 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.935043 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:04.935302 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.935896 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.937034 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.937542 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.937662 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.937805 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.937856 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.937936 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":76,"threads_started":1}
I20260925 22:49:04.938278 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.934122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:40065 (local address 127.0.201.130:46715)
0925 22:49:04.934251 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.934258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.934277 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:49:04.934574 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.934577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.934635 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.934731 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:04.935065 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.935654 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.936801 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.937288 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.937380 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.937419 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.938088 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.938166 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:04.938686 16881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6092341b58004de99f5de4cfe5a29586" candidate_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "028d8019ade7490c8229a79f84f0bf29" is_pre_election: true
I20260925 22:49:04.938833 17019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6092341b58004de99f5de4cfe5a29586" candidate_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" is_pre_election: true
I20260925 22:49:04.939028 16881 raft_consensus.cc:2454] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d92a34f6f7402aa2a9d198faaa7d3e in term 0.
I20260925 22:49:04.939062 17019 raft_consensus.cc:2454] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d92a34f6f7402aa2a9d198faaa7d3e in term 0.
I20260925 22:49:04.939527 17086 leader_election.cc:302] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 028d8019ade7490c8229a79f84f0bf29, d8d92a34f6f7402aa2a9d198faaa7d3e, d9c8f08ebc7d406c952e5ee8295bfa52; no voters:
I20260925 22:49:04.939834 17215 raft_consensus.cc:2781] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:04.939916 17215 raft_consensus.cc:482] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:04.939958 17215 raft_consensus.cc:3037] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:04.940989 17215 raft_consensus.cc:504] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:04.941318 17215 leader_election.cc:288] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [CANDIDATE]: Term 1 election: Requested vote from peers d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715), 028d8019ade7490c8229a79f84f0bf29 (127.0.201.129:37303)
I20260925 22:49:04.941630 17019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6092341b58004de99f5de4cfe5a29586" candidate_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52"
I20260925 22:49:04.941695 16881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6092341b58004de99f5de4cfe5a29586" candidate_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "028d8019ade7490c8229a79f84f0bf29"
I20260925 22:49:04.941746 17019 raft_consensus.cc:3037] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:04.941866 16881 raft_consensus.cc:3037] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:04.942793 17019 raft_consensus.cc:2454] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d92a34f6f7402aa2a9d198faaa7d3e in term 1.
I20260925 22:49:04.942886 16881 raft_consensus.cc:2454] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d92a34f6f7402aa2a9d198faaa7d3e in term 1.
I20260925 22:49:04.943099 17085 leader_election.cc:302] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8d92a34f6f7402aa2a9d198faaa7d3e, d9c8f08ebc7d406c952e5ee8295bfa52; no voters:
I20260925 22:49:04.943270 17215 raft_consensus.cc:2781] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:04.943467 17215 raft_consensus.cc:686] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 1 LEADER]: Becoming Leader. State: Replica: d8d92a34f6f7402aa2a9d198faaa7d3e, State: Running, Role: LEADER
I20260925 22:49:04.943832 17215 consensus_queue.cc:237] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:04.947077 16738 catalog_manager.cc:5495] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131). New cstate: current_term: 1 leader_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } health_report { overall_health: UNKNOWN } } }
I20260925 22:49:04.966406 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.962721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.0.201.131:44729)
0925 22:49:04.962859 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.962863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.962877 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:49:04.962993 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.962996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.963049 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.963142 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:04.963359 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.963973 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.964939 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.965172 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.965264 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.965559 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:04.966013 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:04.966111 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.966221 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.966274 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:04.972694 17019 raft_consensus.cc:1261] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52 [term 1 FOLLOWER]: Refusing update from remote peer d8d92a34f6f7402aa2a9d198faaa7d3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:04.972698 16881 raft_consensus.cc:1261] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Refusing update from remote peer d8d92a34f6f7402aa2a9d198faaa7d3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:04.973481 17218 consensus_queue.cc:1035] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:04.973680 17215 consensus_queue.cc:1035] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:04.981158 17222 mvcc.cc:205] Tried to move back new op lower bound from 7333382328199815168 to 7333382328092708864. Current Snapshot: MvccSnapshot[applied={T|T < 7333382328199815168}]
I20260925 22:49:04.990077 16739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:04.990266 16739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:04.992751 16739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Sending DeleteTablet for 3 replicas of tablet 6092341b58004de99f5de4cfe5a29586
I20260925 22:49:04.993407 17131 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092341b58004de99f5de4cfe5a29586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:04 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:37256
I20260925 22:49:04.993553 16999 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092341b58004de99f5de4cfe5a29586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:04 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:52762
I20260925 22:49:04.993649 16861 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092341b58004de99f5de4cfe5a29586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:04 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:38264
I20260925 22:49:04.994002 17233 tablet_replica.cc:330] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: stopping tablet replica
I20260925 22:49:04.994058 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 16777
I20260925 22:49:04.994351 17233 raft_consensus.cc:2229] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:04.994706 17233 raft_consensus.cc:2258] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:04.995249 17231 tablet_replica.cc:330] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: stopping tablet replica
I20260925 22:49:04.995546 17231 raft_consensus.cc:2229] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:04.995939 17231 raft_consensus.cc:2258] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:04.997467 17231 ts_tablet_manager.cc:1905] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:04.997859 17233 ts_tablet_manager.cc:1905] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:05.000159 17231 ts_tablet_manager.cc:1918] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:05.000237 17231 log.cc:1196] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/6092341b58004de99f5de4cfe5a29586
I20260925 22:49:05.000581 17231 ts_tablet_manager.cc:1939] T 6092341b58004de99f5de4cfe5a29586 P d8d92a34f6f7402aa2a9d198faaa7d3e: Deleting consensus metadata
I20260925 22:49:05.001216 17233 ts_tablet_manager.cc:1918] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:05.001310 17233 log.cc:1196] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/6092341b58004de99f5de4cfe5a29586
I20260925 22:49:05.001645 17233 ts_tablet_manager.cc:1939] T 6092341b58004de99f5de4cfe5a29586 P d9c8f08ebc7d406c952e5ee8295bfa52: Deleting consensus metadata
I20260925 22:49:05.001727 16726 catalog_manager.cc:4841] TS d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729): tablet 6092341b58004de99f5de4cfe5a29586 (table test-table [id=2ab22f97127d44be90c1728e33e2d684]) successfully deleted
W20260925 22:49:05.002152 16725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:37303 (error 108)
I20260925 22:49:05.002780 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 16932
I20260925 22:49:05.002884 16724 catalog_manager.cc:4841] TS d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715): tablet 6092341b58004de99f5de4cfe5a29586 (table test-table [id=2ab22f97127d44be90c1728e33e2d684]) successfully deleted
I20260925 22:49:05.003624 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:05.002959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:37303 (local address 127.0.0.1:38280)
0925 22:49:05.003469 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:05.003542 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432}
W20260925 22:49:05.004287 16725 catalog_manager.cc:4584] TS 028d8019ade7490c8229a79f84f0bf29 (127.0.201.129:37303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6092341b58004de99f5de4cfe5a29586: Network error: Client connection negotiation failed: client connection to 127.0.201.129:37303: connect: Connection refused (error 111)
I20260925 22:49:05.008993 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 17068
I20260925 22:49:05.014730 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 16706
I20260925 22:49:25.021288 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37517
--webserver_interface=127.0.201.190
--webserver_port=46511
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:37517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:25.133971 17235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:25.134225 17235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:25.134264 17235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:25.138238 17235 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:49:25.138289 17235 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:49:25.138304 17235 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:49:25.138319 17235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:25.138337 17235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:25.138407 17235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:25.138435 17235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:25.138468 17235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:25.143652 17235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:37517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:37517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:25.144731 17235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:25.145787 17235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:25.154608 17241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:25.154654 17235 server_base.cc:1034] running on GCE node
W20260925 22:49:25.154618 17240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:25.154618 17243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:25.155458 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:25.156600 17235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:25.157765 17235 hybrid_clock.cc:648] HybridClock initialized: now 1790376565157728 us; error 49 us; skew 500 ppm
Sep 25 22:49:25 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:25.168010 17235 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:49:25.168987 17235 webserver.cc:460] Webserver started at http://127.0.201.190:46511/ using document root <none> and password file <none>
I20260925 22:49:25.169607 17235 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:25.169701 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:25.173758 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:25.176182 17250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:25.177570 17235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:25.177716 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "1facd2cfc739458db9d7a62a276c0fef"
format_stamp: "Formatted at 2026-09-25 22:48:42 on dist-test-slave-vdtf"
server_key: "6bf12e35920326c8c0abf84718575c46"
server_key_iv: "1c324b65749d926b7008eeb612f38b4e"
server_key_version: "encryptionkey@0"
I20260925 22:49:25.178283 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:25.205549 17235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:25.208595 17235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:25.208886 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:25.215718 17235 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:37517
I20260925 22:49:25.216046 17302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:37517 every 8 connection(s)
I20260925 22:49:25.216867 17235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:25.217844 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 17235
I20260925 22:49:25.218286 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:37303
--local_ip_for_outbound_sockets=127.0.201.129
--tserver_master_addrs=127.0.201.190:37517
--webserver_port=36003
--webserver_interface=127.0.201.129
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:25.221922 17303 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:49:25.223595 17303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Bootstrap starting.
I20260925 22:49:25.240872 17303 log.cc:824] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:25.252067 17303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:49:25.252491 17303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Bootstrap complete.
I20260925 22:49:25.256453 17303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:49:25.256848 17303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1facd2cfc739458db9d7a62a276c0fef, State: Initialized, Role: FOLLOWER
I20260925 22:49:25.257355 17303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:49:25.257501 17303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:25.257584 17303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:25.257695 17303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:49:25.259569 17303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:49:25.259930 17303 leader_election.cc:302] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1facd2cfc739458db9d7a62a276c0fef; no voters:
I20260925 22:49:25.260241 17303 leader_election.cc:288] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:49:25.260417 17307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:49:25.260748 17307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [term 3 LEADER]: Becoming Leader. State: Replica: 1facd2cfc739458db9d7a62a276c0fef, State: Running, Role: LEADER
I20260925 22:49:25.261202 17307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } }
I20260925 22:49:25.261497 17303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:25.262643 17308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1facd2cfc739458db9d7a62a276c0fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } } }
I20260925 22:49:25.262773 17308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:25.263134 17309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1facd2cfc739458db9d7a62a276c0fef. Latest consensus state: current_term: 3 leader_uuid: "1facd2cfc739458db9d7a62a276c0fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1facd2cfc739458db9d7a62a276c0fef" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 37517 } } }
I20260925 22:49:25.263265 17309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:25.263581 17316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:25.267904 17316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:25.268429 17316 catalog_manager.cc:1259] Loaded cluster ID: 8e98aab1064d463696c76819421d324d
I20260925 22:49:25.268504 17316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:25.270287 17316 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:25.270886 17316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Loaded TSK: 0
I20260925 22:49:25.270972 17316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1facd2cfc739458db9d7a62a276c0fef: Loaded TSK: 1
F20260925 22:49:25.271574 17316 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0e89ea59ad google::LogMessage::Fail() at ??:0
@ 0x7f0e89ea888a google::LogMessage::SendToLog() at ??:0
@ 0x7f0e89ea54de google::LogMessage::Flush() at ??:0
@ 0x7f0e89ea6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0e8cb6918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0e8cb61f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0e9480a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0e947e499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0e947e5178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f0e94826df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f0e9484582b std::function<>::operator()() at ??:0
@ 0x7f0e947e4b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0e947e5f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0e947e2579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0e94826640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0e8c0f73d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0e8c0f7c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0e8c0f93f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0e8c0e8512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0e8d72c6db start_thread at ??:0
@ 0x7f0e889a361f clone at ??:0
W20260925 22:49:25.350008 17305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:25.350231 17305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:25.350273 17305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:25.354609 17305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:25.354652 17305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:25.354709 17305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:49:25.360495 17305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:37303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=36003
--tserver_master_addrs=127.0.201.190:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:25.361531 17305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:25.362470 17305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:25.372017 17351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:25.372054 17353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:25.372036 17350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:25.372282 17305 server_base.cc:1034] running on GCE node
I20260925 22:49:25.373028 17305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:25.373769 17305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:25.374962 17305 hybrid_clock.cc:648] HybridClock initialized: now 1790376565374938 us; error 35 us; skew 500 ppm
Sep 25 22:49:25 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:25.378356 17305 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:49:25.379465 17305 webserver.cc:460] Webserver started at http://127.0.201.129:36003/ using document root <none> and password file <none>
I20260925 22:49:25.380198 17305 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:25.380290 17305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:25.385521 17305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:25.388265 17360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:25.389834 17305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:25.389962 17305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "028d8019ade7490c8229a79f84f0bf29"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "bb603a8e20f77fea6234ca7a16931224"
server_key_iv: "8270cc92bf22e150afd22fa0fd83225b"
server_key_version: "encryptionkey@0"
I20260925 22:49:25.390652 17305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:25.407891 17305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:25.411790 17305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:25.412004 17305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:25.412708 17305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:25.414038 17367 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:49:25.418159 17305 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:49:25.418252 17305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260925 22:49:25.418318 17305 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:49:25.420662 17305 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:49:25.420720 17305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:25.421098 17367 tablet_bootstrap.cc:492] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Bootstrap starting.
I20260925 22:49:25.432062 17305 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:37303
I20260925 22:49:25.432502 17474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:37303 every 8 connection(s)
I20260925 22:49:25.433358 17305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:25.435034 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 17305
I20260925 22:49:25.435510 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:46715
--local_ip_for_outbound_sockets=127.0.201.130
--tserver_master_addrs=127.0.201.190:37517
--webserver_port=45939
--webserver_interface=127.0.201.130
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:25.446679 17367 log.cc:824] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:25.455873 17367 tablet_bootstrap.cc:492] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:49:25.456300 17367 tablet_bootstrap.cc:492] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Bootstrap complete.
I20260925 22:49:25.457688 17367 ts_tablet_manager.cc:1397] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Time spent bootstrapping tablet: real 0.037s user 0.030s sys 0.000s
I20260925 22:49:25.460955 17367 raft_consensus.cc:348] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:25.461514 17367 raft_consensus.cc:729] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028d8019ade7490c8229a79f84f0bf29, State: Initialized, Role: FOLLOWER
I20260925 22:49:25.462031 17367 consensus_queue.cc:260] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:25.463065 17367 ts_tablet_manager.cc:1428] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260925 22:49:25.516629 17477 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:25.435815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:43229)
0925 22:49:25.437341 (+ 1526us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:25.437408 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:25.438528 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:25.515865 (+ 77337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1269,"thread_start_us":148,"threads_started":1}
W20260925 22:49:25.517168 17475 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:37517 (0 consecutive failures): Network error: Failed to ping master at 127.0.201.190:37517: Client connection negotiation failed: client connection to 127.0.201.190:37517: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:49:25.517971 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:25.517713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:56961)
0925 22:49:25.517855 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:25.517917 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:49:25.518641 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:25.518439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:44513)
0925 22:49:25.518544 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:25.518597 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:49:25.518826 17475 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:49:25.555320 17478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:25.555621 17478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:25.555744 17478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:25.559824 17478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:25.559902 17478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:25.559993 17478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:49:25.565235 17478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:46715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=45939
--tserver_master_addrs=127.0.201.190:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:25.566282 17478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:25.567365 17478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:25.577296 17485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:25.577301 17488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:25.577389 17478 server_base.cc:1034] running on GCE node
W20260925 22:49:25.577291 17486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:25.578017 17478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:25.578630 17478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:25.579811 17478 hybrid_clock.cc:648] HybridClock initialized: now 1790376565579796 us; error 28 us; skew 500 ppm
Sep 25 22:49:25 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:25.582620 17478 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:49:25.583499 17478 webserver.cc:460] Webserver started at http://127.0.201.130:45939/ using document root <none> and password file <none>
I20260925 22:49:25.584148 17478 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:25.584242 17478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:25.588115 17478 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:25.590215 17495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:25.591388 17478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:25.591524 17478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "d9c8f08ebc7d406c952e5ee8295bfa52"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "5a46cf996036a979802b363263021244"
server_key_iv: "f1c9d1799a8bee0cdeabb66ec7184a43"
server_key_version: "encryptionkey@0"
I20260925 22:49:25.592130 17478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:25.616442 17478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:25.620103 17478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:25.620322 17478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:25.620957 17478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:25.621802 17478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:25.621874 17478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:25.621948 17478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:25.621989 17478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:25.630959 17478 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:46715
I20260925 22:49:25.630990 17608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:46715 every 8 connection(s)
I20260925 22:49:25.631953 17478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:25.634240 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 17478
I20260925 22:49:25.634680 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:44729
--local_ip_for_outbound_sockets=127.0.201.131
--tserver_master_addrs=127.0.201.190:37517
--webserver_port=33119
--webserver_interface=127.0.201.131
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:25.635599 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:25.634041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:46021)
0925 22:49:25.634749 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:25.634887 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":148,"threads_started":1}
W20260925 22:49:25.636268 17609 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:37517 (0 consecutive failures): Network error: Failed to ping master at 127.0.201.190:37517: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
I20260925 22:49:25.636978 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:25.636712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:55821)
0925 22:49:25.636864 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:25.636922 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 22:49:25.637599 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:25.637388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:59131)
0925 22:49:25.637496 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:25.637540 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:49:25.637781 17609 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:49:25.744556 17612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:25.744823 17612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:25.744866 17612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:25.748874 17612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:25.748930 17612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:25.748988 17612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:49:25.754179 17612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:44729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=33119
--tserver_master_addrs=127.0.201.190:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:25.755220 17612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:25.756273 17612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:25.765900 17617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:25.765965 17612 server_base.cc:1034] running on GCE node
W20260925 22:49:25.765916 17618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:25.765918 17620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:25.766628 17612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:25.767254 17612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:25.768432 17612 hybrid_clock.cc:648] HybridClock initialized: now 1790376565768401 us; error 45 us; skew 500 ppm
Sep 25 22:49:25 dist-test-slave-vdtf krb5kdc[16158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:25.771003 17612 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:49:25.771862 17612 webserver.cc:460] Webserver started at http://127.0.201.131:33119/ using document root <none> and password file <none>
I20260925 22:49:25.772383 17612 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:25.772440 17612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:25.776247 17612 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 22:49:25.778182 17627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:25.779383 17612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:25.779474 17612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-vdtf"
server_key: "847b346fc4a11e19e01cf29dd4c12598"
server_key_iv: "efe1e218240f4686ab51ca00dfc8d7a1"
server_key_version: "encryptionkey@0"
I20260925 22:49:25.780035 17612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:25.800812 17612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:25.803663 17612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:25.803844 17612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:25.804417 17612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:25.805173 17612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:25.805217 17612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:25.805256 17612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:25.805271 17612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:25.814415 17612 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:44729
I20260925 22:49:25.814435 17740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:44729 every 8 connection(s)
I20260925 22:49:25.815387 17612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:25.819162 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:25.817466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:38447)
0925 22:49:25.818366 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:25.818476 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":525,"threads_started":1}
W20260925 22:49:25.819605 17741 heartbeater.cc:644] Failed to heartbeat to 127.0.201.190:37517 (0 consecutive failures): Network error: Failed to ping master at 127.0.201.190:37517: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
I20260925 22:49:25.820170 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:25.819973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:40671)
0925 22:49:25.820075 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:25.820120 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:49:25.820749 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:25.820573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:34403)
0925 22:49:25.820665 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:25.820704 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:49:25.820976 17741 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:49:25.821203 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 17612
I20260925 22:49:26.519835 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:26.519379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:50503)
0925 22:49:26.519616 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:26.519668 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260925 22:49:26.638671 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:26.638292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:37229)
0925 22:49:26.638519 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:26.638571 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260925 22:49:26.821962 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:26.821555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:42697)
0925 22:49:26.821817 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:26.821867 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":90,"threads_started":1}
I20260925 22:49:27.050132 17748 raft_consensus.cc:482] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:27.050407 17748 raft_consensus.cc:504] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:27.051558 17748 leader_election.cc:288] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729), d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715)
Sep 25 22:49:27 dist-test-slave-vdtf krb5kdc[16158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.131@KRBTEST.COM
Sep 25 22:49:27 dist-test-slave-vdtf krb5kdc[16158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.130@KRBTEST.COM
I20260925 22:49:27.066181 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:27.051807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:46715 (local address 127.0.201.129:49351)
0925 22:49:27.052070 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:27.052086 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:27.052188 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:27.053898 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:27.053904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:27.054370 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:27.054976 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:27.054996 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:27.056327 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:27.056335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:27.057314 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:27.057323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:27.058464 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:27.058471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:27.058616 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:27.059628 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:27.059659 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:27.061333 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:27.063543 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:27.063549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:27.063561 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:27.063929 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:27.064361 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:27.064364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:27.064365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:27.064522 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:27.065130 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:27.065136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:27.065383 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 22:49:27.065598 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 22:49:27.066004 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":227,"thread_start_us":60,"threads_started":1}
I20260925 22:49:27.066180 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:27.051831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:44729 (local address 127.0.201.129:37577)
0925 22:49:27.052370 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:27.052383 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:27.052473 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:27.054599 (+ 2126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:27.054602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:27.054781 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:27.054996 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:27.055003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:27.056990 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:27.056993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:27.057664 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:27.057671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:27.058370 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:27.058374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:27.058543 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:27.059324 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:27.059340 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:27.060988 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:27.063836 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:27.063839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:27.063841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:27.064084 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:27.064448 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:27.064451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:27.064453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:27.064557 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:27.065383 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:27.065386 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:27.065454 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 22:49:27.065598 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 22:49:27.065952 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":53,"thread_start_us":61,"threads_started":1}
I20260925 22:49:27.067503 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:27.052012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:49351 (local address 127.0.201.130:46715)
0925 22:49:27.052431 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:27.052456 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:27.052503 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:49:27.053198 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:27.053204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:27.053534 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:27.053862 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:27.055161 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:27.056193 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:27.057438 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:27.058304 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:27.059001 (+ 697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:27.061522 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:27.061526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:27.061537 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:27.061563 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:27.063416 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:27.064075 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:27.064083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:27.064091 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:27.064252 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:27.064620 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:27.064624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:27.064626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:27.065012 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:27.065155 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:27.066906 (+ 1751us) server_negotiation.cc:299] Negotiation successful
0925 22:49:27.067243 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":73,"threads_started":1}
I20260925 22:49:27.067822 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:27.052277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:37577 (local address 127.0.201.131:44729)
0925 22:49:27.052711 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:27.052738 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:27.052782 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:49:27.053899 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:27.053910 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:27.054335 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:27.054669 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:27.055127 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:27.056844 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:27.057778 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:27.058243 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:27.058428 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:27.061284 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:27.061294 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:27.061310 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:27.061348 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:27.063723 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:27.064197 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:27.064202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:27.064207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:27.064354 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:27.064695 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:27.064700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:27.064704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:27.065268 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:27.065405 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:27.065745 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 22:49:27.066060 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":96,"threads_started":1}
W20260925 22:49:27.069939 17362 leader_election.cc:341] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715): Not found: Tablet not found: 6092341b58004de99f5de4cfe5a29586
W20260925 22:49:27.070529 17364 leader_election.cc:341] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729): Not found: Tablet not found: 6092341b58004de99f5de4cfe5a29586
I20260925 22:49:27.070641 17364 leader_election.cc:302] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 028d8019ade7490c8229a79f84f0bf29; no voters: d8d92a34f6f7402aa2a9d198faaa7d3e, d9c8f08ebc7d406c952e5ee8295bfa52
I20260925 22:49:27.071108 17748 raft_consensus.cc:2726] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:49:27.520846 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:27.520549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:52819)
0925 22:49:27.520687 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:27.520763 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 22:49:27.639997 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:27.639492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:43429)
0925 22:49:27.639843 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:27.639904 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":74,"threads_started":1}
I20260925 22:49:27.823133 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:27.822733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:54691)
0925 22:49:27.822979 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:27.823045 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260925 22:49:28.522004 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:28.521581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:58459)
0925 22:49:28.521852 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:28.521910 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260925 22:49:28.631639 17756 raft_consensus.cc:482] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:28.631819 17756 raft_consensus.cc:504] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:28.632274 17756 leader_election.cc:288] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729), d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715)
W20260925 22:49:28.633141 17362 leader_election.cc:341] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715): Not found: Tablet not found: 6092341b58004de99f5de4cfe5a29586
W20260925 22:49:28.633279 17364 leader_election.cc:341] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729): Not found: Tablet not found: 6092341b58004de99f5de4cfe5a29586
I20260925 22:49:28.633366 17364 leader_election.cc:302] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 028d8019ade7490c8229a79f84f0bf29; no voters: d8d92a34f6f7402aa2a9d198faaa7d3e, d9c8f08ebc7d406c952e5ee8295bfa52
I20260925 22:49:28.633512 17756 raft_consensus.cc:2726] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:49:28.641049 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:28.640653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:53491)
0925 22:49:28.640919 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:28.640972 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":118,"threads_started":1}
I20260925 22:49:28.824415 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:28.823906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:39755)
0925 22:49:28.824251 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:28.824308 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":71,"threads_started":1}
I20260925 22:49:29.523402 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:29.522992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:55611)
0925 22:49:29.523250 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:29.523309 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260925 22:49:29.642248 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:29.641816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:51603)
0925 22:49:29.642079 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:29.642143 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260925 22:49:29.825556 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:29.825143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:53829)
0925 22:49:29.825398 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:29.825456 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260925 22:49:30.423448 17762 raft_consensus.cc:482] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:30.423589 17762 raft_consensus.cc:504] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:30.424154 17762 leader_election.cc:288] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729), d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715)
W20260925 22:49:30.424904 17364 leader_election.cc:341] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729): Not found: Tablet not found: 6092341b58004de99f5de4cfe5a29586
W20260925 22:49:30.425001 17362 leader_election.cc:341] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715): Not found: Tablet not found: 6092341b58004de99f5de4cfe5a29586
I20260925 22:49:30.425127 17362 leader_election.cc:302] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 028d8019ade7490c8229a79f84f0bf29; no voters: d8d92a34f6f7402aa2a9d198faaa7d3e, d9c8f08ebc7d406c952e5ee8295bfa52
I20260925 22:49:30.425297 17762 raft_consensus.cc:2726] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:49:30.524796 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:30.524345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:53845)
0925 22:49:30.524638 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:30.524694 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
I20260925 22:49:30.643453 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:30.643062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:42209)
0925 22:49:30.643307 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:30.643365 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":78,"threads_started":1}
I20260925 22:49:30.826901 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:30.826409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:40371)
0925 22:49:30.826703 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:30.826769 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
I20260925 22:49:31.525872 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.525439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:54149)
0925 22:49:31.525702 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:31.525779 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260925 22:49:31.644624 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.644267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:39965)
0925 22:49:31.644487 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:31.644535 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260925 22:49:31.827978 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.827560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:57217)
0925 22:49:31.827838 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:31.827890 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260925 22:49:32.527158 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:32.526690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:41619)
0925 22:49:32.526956 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:32.527011 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260925 22:49:32.645854 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:32.645362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:57543)
0925 22:49:32.645697 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:32.645760 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260925 22:49:32.829044 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:32.828626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:41951)
0925 22:49:32.828893 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:32.828943 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":69,"threads_started":1}
I20260925 22:49:33.073823 17772 raft_consensus.cc:482] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:33.073961 17772 raft_consensus.cc:504] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:33.074422 17772 leader_election.cc:288] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729), d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715)
W20260925 22:49:33.075222 17364 leader_election.cc:341] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729): Not found: Tablet not found: 6092341b58004de99f5de4cfe5a29586
W20260925 22:49:33.075338 17362 leader_election.cc:341] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715): Not found: Tablet not found: 6092341b58004de99f5de4cfe5a29586
I20260925 22:49:33.075459 17362 leader_election.cc:302] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 028d8019ade7490c8229a79f84f0bf29; no voters: d8d92a34f6f7402aa2a9d198faaa7d3e, d9c8f08ebc7d406c952e5ee8295bfa52
I20260925 22:49:33.075644 17772 raft_consensus.cc:2726] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:49:33.528458 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.528047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:52873)
0925 22:49:33.528308 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.528363 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":72,"threads_started":1}
I20260925 22:49:33.647017 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.646617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:57579)
0925 22:49:33.646860 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.646923 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
I20260925 22:49:33.830178 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.829760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:53111)
0925 22:49:33.830022 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.830075 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260925 22:49:34.529619 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.529225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:35255)
0925 22:49:34.529471 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:34.529528 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260925 22:49:34.648257 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.647812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:60221)
0925 22:49:34.648087 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:34.648158 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260925 22:49:34.831341 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.830874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:60331)
0925 22:49:34.831147 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:34.831203 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":74,"threads_started":1}
I20260925 22:49:35.530980 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:35.530457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:57013)
0925 22:49:35.530805 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:35.530882 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":71,"threads_started":1}
I20260925 22:49:35.649441 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:35.649053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:38917)
0925 22:49:35.649289 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:35.649345 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260925 22:49:35.832527 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:35.832108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:47409)
0925 22:49:35.832361 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:35.832420 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260925 22:49:36.532136 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.531754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:54593)
0925 22:49:36.531987 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.532041 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260925 22:49:36.650600 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.650195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:44503)
0925 22:49:36.650444 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.650508 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260925 22:49:36.833781 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.833311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:49929)
0925 22:49:36.833629 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.833687 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":95,"threads_started":1}
I20260925 22:49:37.533362 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.532921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:52341)
0925 22:49:37.533206 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.533267 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
I20260925 22:49:37.651988 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.651493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:51053)
0925 22:49:37.651831 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.651891 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":79,"threads_started":1}
I20260925 22:49:37.835002 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.834568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:50935)
0925 22:49:37.834856 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.834912 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":74,"threads_started":1}
I20260925 22:49:38.534626 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:38.534191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:34029)
0925 22:49:38.534458 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:38.534520 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260925 22:49:38.583323 17789 raft_consensus.cc:482] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:38.583452 17789 raft_consensus.cc:504] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d92a34f6f7402aa2a9d198faaa7d3e" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44729 } } peers { permanent_uuid: "d9c8f08ebc7d406c952e5ee8295bfa52" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 46715 } } peers { permanent_uuid: "028d8019ade7490c8229a79f84f0bf29" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 37303 } }
I20260925 22:49:38.583964 17789 leader_election.cc:288] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729), d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715)
W20260925 22:49:38.584802 17364 leader_election.cc:341] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8d92a34f6f7402aa2a9d198faaa7d3e (127.0.201.131:44729): Not found: Tablet not found: 6092341b58004de99f5de4cfe5a29586
W20260925 22:49:38.584939 17362 leader_election.cc:341] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9c8f08ebc7d406c952e5ee8295bfa52 (127.0.201.130:46715): Not found: Tablet not found: 6092341b58004de99f5de4cfe5a29586
I20260925 22:49:38.585042 17362 leader_election.cc:302] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 028d8019ade7490c8229a79f84f0bf29; no voters: d8d92a34f6f7402aa2a9d198faaa7d3e, d9c8f08ebc7d406c952e5ee8295bfa52
I20260925 22:49:38.585220 17789 raft_consensus.cc:2726] T 6092341b58004de99f5de4cfe5a29586 P 028d8019ade7490c8229a79f84f0bf29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:49:38.653172 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:38.652754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:56907)
0925 22:49:38.653014 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:38.653074 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260925 22:49:38.836405 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:38.835947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:34505)
0925 22:49:38.836233 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:38.836301 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":124,"threads_started":1}
I20260925 22:49:39.535878 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:39.535436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:34711)
0925 22:49:39.535719 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:39.535785 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260925 22:49:39.654295 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:39.653907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:57751)
0925 22:49:39.654148 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:39.654203 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260925 22:49:39.837724 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:39.837203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.131:52217)
0925 22:49:39.837511 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:39.837585 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
I20260925 22:49:40.537163 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:40.536786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.129:57843)
0925 22:49:40.537013 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:40.537066 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260925 22:49:40.655551 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:40.655141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:37517 (local address 127.0.201.130:41289)
0925 22:49:40.655401 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:40.655457 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.190:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
I20260925 22:49:40.823256 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 17305
I20260925 22:49:40.829093 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 17478
I20260925 22:49:40.835821 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 17612
2026-09-25T22:49:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59185 ms)
[----------] 30 tests from SecurityITest (239215 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:40 dist-test-slave-vdtf krb5kdc[17799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:40 dist-test-slave-vdtf krb5kdc[17799](info): set up 2 sockets
Sep 25 22:49:40 dist-test-slave-vdtf krb5kdc[17799](info): commencing operation
krb5kdc: starting...
W20260925 22:49:41.895453 806 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:49:41 dist-test-slave-vdtf krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:49:42.015434 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:33365
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:42.123286 17812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:42.123566 17812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:42.123641 17812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:42.127741 17812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:42.127820 17812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:42.127857 17812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:42.127911 17812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:42.127955 17812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:42.133080 17812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:42.134104 17812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:42.135149 17812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:42.143333 17817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.143333 17820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.143376 17818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:42.143571 17812 server_base.cc:1034] running on GCE node
I20260925 22:49:42.144285 17812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:42.145417 17812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:42.146586 17812 hybrid_clock.cc:648] HybridClock initialized: now 1790376582146566 us; error 33 us; skew 500 ppm
Sep 25 22:49:42 dist-test-slave-vdtf krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:42.149296 17812 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:49:42.150193 17812 webserver.cc:460] Webserver started at http://127.0.201.190:41681/ using document root <none> and password file <none>
I20260925 22:49:42.150738 17812 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:42.150828 17812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:42.151062 17812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:42.153091 17812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "106aea0b028f4f6cb25d7b8346db94c7"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "47eaf979d8d4b1399f0d7520773dce37"
server_key_iv: "328dc5b4576ca2e20415c295790e5e6d"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.153636 17812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "106aea0b028f4f6cb25d7b8346db94c7"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "47eaf979d8d4b1399f0d7520773dce37"
server_key_iv: "328dc5b4576ca2e20415c295790e5e6d"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.157155 17812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:49:42.159476 17827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.160799 17812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:42.160935 17812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "106aea0b028f4f6cb25d7b8346db94c7"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "47eaf979d8d4b1399f0d7520773dce37"
server_key_iv: "328dc5b4576ca2e20415c295790e5e6d"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.161051 17812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:42.177109 17812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:42.179803 17812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:42.180002 17812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:42.186846 17812 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:33365
I20260925 22:49:42.186857 17879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:33365 every 8 connection(s)
I20260925 22:49:42.187783 17812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:42.190442 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:42.190840 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 17812
I20260925 22:49:42.191011 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:42.191318 806 external_mini_cluster.cc:1428] Setting key 6dc0d353f2fe9b13b5275f0a5d17e41d
I20260925 22:49:42.197137 17880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7: Bootstrap starting.
I20260925 22:49:42.199852 17880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7: Neither blocks nor log segments found. Creating new log.
Sep 25 22:49:42 dist-test-slave-vdtf krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:42.200601 17880 log.cc:824] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:42.202387 17880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7: No bootstrap required, opened a new log
I20260925 22:49:42.205233 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.192605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.0.201.190:33365)
0925 22:49:42.192990 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.193005 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.193039 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:49:42.193666 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.193682 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.193991 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.194260 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:42.195169 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.196199 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.197350 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.197804 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.197953 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.200725 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:42.200731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:42.200746 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:42.200775 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:42.202539 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.203014 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.203019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.203023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.203138 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.203417 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.203419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.203420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.203834 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:42.203972 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.204270 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.204542 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":129,"threads_started":1}
I20260925 22:49:42.206398 17880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106aea0b028f4f6cb25d7b8346db94c7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33365 } }
I20260925 22:49:42.206655 17880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:42.206710 17880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106aea0b028f4f6cb25d7b8346db94c7, State: Initialized, Role: FOLLOWER
I20260925 22:49:42.207136 17880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106aea0b028f4f6cb25d7b8346db94c7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33365 } }
I20260925 22:49:42.207306 17880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:42.207370 17880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:42.207485 17880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:42.208349 17880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106aea0b028f4f6cb25d7b8346db94c7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33365 } }
I20260925 22:49:42.208674 17880 leader_election.cc:302] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 106aea0b028f4f6cb25d7b8346db94c7; no voters:
I20260925 22:49:42.208985 17880 leader_election.cc:288] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:42.209096 17885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:42.209307 17885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [term 1 LEADER]: Becoming Leader. State: Replica: 106aea0b028f4f6cb25d7b8346db94c7, State: Running, Role: LEADER
I20260925 22:49:42.209589 17885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106aea0b028f4f6cb25d7b8346db94c7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33365 } }
I20260925 22:49:42.210134 17880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:42.211181 17886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "106aea0b028f4f6cb25d7b8346db94c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106aea0b028f4f6cb25d7b8346db94c7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33365 } } }
I20260925 22:49:42.211308 17886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:42.211702 17894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:42.212543 17887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 106aea0b028f4f6cb25d7b8346db94c7. Latest consensus state: current_term: 1 leader_uuid: "106aea0b028f4f6cb25d7b8346db94c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106aea0b028f4f6cb25d7b8346db94c7" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 33365 } } }
I20260925 22:49:42.212639 17887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:42.214757 17894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:42.220468 17894 catalog_manager.cc:1347] Generated new cluster ID: 42145c5a65914cd8acefee16c5ad9541
I20260925 22:49:42.220549 17894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:42.244881 17894 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:42.245868 17894 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:42.251976 17894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7: Generated new TSK 0
I20260925 22:49:42.252794 17894 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:49:42.290968 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33365
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:42.400177 17906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:42.400426 17906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:42.400501 17906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:42.407275 17906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:42.407342 17906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:42.407402 17906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:49:42.413070 17906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:42.414143 17906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:42.415226 17906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:42.424857 17912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.424927 17911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.424857 17914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:42.425259 17906 server_base.cc:1034] running on GCE node
I20260925 22:49:42.425757 17906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:42.426550 17906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:42.427755 17906 hybrid_clock.cc:648] HybridClock initialized: now 1790376582427740 us; error 31 us; skew 500 ppm
Sep 25 22:49:42 dist-test-slave-vdtf krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:42.431321 17906 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:49:42.432525 17906 webserver.cc:460] Webserver started at http://127.0.201.129:39389/ using document root <none> and password file <none>
I20260925 22:49:42.433216 17906 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:42.433290 17906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:42.433523 17906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:42.436460 17906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "679a90271eda4aaf821d281c07de2ad8"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "1aea3e11d7913f66b61a3faf2471ccd7"
server_key_iv: "4946efa80e73662a263e5d8d8206b9e5"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.437229 17906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "679a90271eda4aaf821d281c07de2ad8"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "1aea3e11d7913f66b61a3faf2471ccd7"
server_key_iv: "4946efa80e73662a263e5d8d8206b9e5"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.442407 17906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:42.445669 17921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.447152 17906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:42.447270 17906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "679a90271eda4aaf821d281c07de2ad8"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "1aea3e11d7913f66b61a3faf2471ccd7"
server_key_iv: "4946efa80e73662a263e5d8d8206b9e5"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.447393 17906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:42.471942 17906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:42.475060 17906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:42.475206 17906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:42.475783 17906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:42.476564 17906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:42.476609 17906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.476648 17906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:42.476670 17906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.485277 17906 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:39503
I20260925 22:49:42.485373 18034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:39503 every 8 connection(s)
I20260925 22:49:42.486155 17906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:42.489953 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 17906
I20260925 22:49:42.490101 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:42.490396 806 external_mini_cluster.cc:1428] Setting key 30c0143bfdbb154c9c3015850e5be6fd
Sep 25 22:49:42 dist-test-slave-vdtf krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:49:42.500751 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.487924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:48429 (local address 127.0.201.190:33365)
0925 22:49:42.488071 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.488076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.489060 (+ 984us) server_negotiation.cc:407] Connection header received
0925 22:49:42.490581 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.490583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.490630 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.490705 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:42.492112 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.492630 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.494086 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.494327 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.494455 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.497138 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:42.497143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:42.497150 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:42.497184 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:42.498809 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.499279 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.499286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.499288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.499385 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.499725 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.499729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.499731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.499950 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:42.500053 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.500530 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.500632 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:49:42.501552 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.488373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33365 (local address 127.0.201.129:48429)
0925 22:49:42.488902 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:42.488944 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:42.490335 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:42.490872 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:42.490879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:42.491309 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:42.491963 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.491984 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.492758 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.492765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.493955 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.493964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.494518 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.494521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.494662 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.495253 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:42.495268 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:42.496967 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:42.498921 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:42.498925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:42.498936 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:42.499178 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:42.499484 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:42.499486 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:42.499488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:42.499616 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:42.500053 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:42.500058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:42.500312 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 22:49:42.500465 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:49:42.500712 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:49:42.502758 18035 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33365
I20260925 22:49:42.502985 18035 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:42.503466 18035 heartbeater.cc:511] Master 127.0.201.190:33365 requested a full tablet report, sending...
I20260925 22:49:42.505255 17844 ts_manager.cc:194] Registered new tserver with Master: 679a90271eda4aaf821d281c07de2ad8 (127.0.201.129:39503)
I20260925 22:49:42.506371 17844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:48429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:49:42.516885 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33365
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:42.623555 18040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:42.623854 18040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:42.623899 18040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:42.627861 18040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:42.627910 18040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:42.627966 18040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:49:42.633103 18040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:42.634083 18040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:42.635053 18040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:42.644672 18048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.644672 18046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.644666 18045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:42.645056 18040 server_base.cc:1034] running on GCE node
I20260925 22:49:42.645442 18040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:42.646170 18040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:42.647328 18040 hybrid_clock.cc:648] HybridClock initialized: now 1790376582647319 us; error 29 us; skew 500 ppm
Sep 25 22:49:42 dist-test-slave-vdtf krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:42.650198 18040 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:49:42.651108 18040 webserver.cc:460] Webserver started at http://127.0.201.130:42353/ using document root <none> and password file <none>
I20260925 22:49:42.651666 18040 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:42.651808 18040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:42.652109 18040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:42.654192 18040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "5c968f1357204a0880373fba0731d906"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "f88585575befdca0279e6d3533affa45"
server_key_iv: "3a4d227e3ce41bf3bdda3536918d3e02"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.654752 18040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "5c968f1357204a0880373fba0731d906"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "f88585575befdca0279e6d3533affa45"
server_key_iv: "3a4d227e3ce41bf3bdda3536918d3e02"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.658452 18040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:49:42.660789 18055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.661945 18040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:42.662081 18040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "5c968f1357204a0880373fba0731d906"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "f88585575befdca0279e6d3533affa45"
server_key_iv: "3a4d227e3ce41bf3bdda3536918d3e02"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.662184 18040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:42.690593 18040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:42.693364 18040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:42.693562 18040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:42.694136 18040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:42.694913 18040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:42.694983 18040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.695051 18040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:42.695106 18040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.703392 18040 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:37799
I20260925 22:49:42.703419 18168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:37799 every 8 connection(s)
I20260925 22:49:42.704268 18040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:42.712996 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 18040
I20260925 22:49:42.713141 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:42.713514 806 external_mini_cluster.cc:1428] Setting key d2afaf7d71c5f68a0db4471f1985d06f
Sep 25 22:49:42 dist-test-slave-vdtf krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:42.718075 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.706043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:43325 (local address 127.0.201.190:33365)
0925 22:49:42.706185 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.706188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.707146 (+ 958us) server_negotiation.cc:407] Connection header received
0925 22:49:42.708541 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.708544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.708591 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.708672 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:42.709746 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.710386 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.711581 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.711920 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.712011 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.714546 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:42.714549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:42.714550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:42.714580 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:42.716149 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.716643 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.716649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.716651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.716730 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.717062 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.717066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.717068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.717242 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:42.717346 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.717798 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.717912 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:49:42.718825 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.706378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33365 (local address 127.0.201.130:43325)
0925 22:49:42.706958 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:42.707021 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:42.708327 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:42.708837 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:42.708843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:42.709175 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:42.709620 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.709633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.710489 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.710492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.711472 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.711481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.712047 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.712050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.712182 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.712686 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:42.712701 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:42.714382 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:42.716278 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:42.716283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:42.716297 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:42.716541 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:42.716828 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:42.716831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:42.716832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:42.716966 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:42.717344 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:42.717349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:42.717578 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 22:49:42.717755 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 22:49:42.718044 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":146,"threads_started":1}
I20260925 22:49:42.720088 18169 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33365
I20260925 22:49:42.720314 18169 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:42.720768 18169 heartbeater.cc:511] Master 127.0.201.190:33365 requested a full tablet report, sending...
I20260925 22:49:42.721789 17844 ts_manager.cc:194] Registered new tserver with Master: 5c968f1357204a0880373fba0731d906 (127.0.201.130:37799)
I20260925 22:49:42.722332 17844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:43325
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:49:42.740857 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33365
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:42.854830 18174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:42.855093 18174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:42.855134 18174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:42.859084 18174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:42.859138 18174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:42.859194 18174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:49:42.864462 18174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:42.865545 18174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:42.866549 18174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:42.875866 18180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.875874 18182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.875950 18179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:42.875923 18174 server_base.cc:1034] running on GCE node
I20260925 22:49:42.876425 18174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:42.877017 18174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:42.878201 18174 hybrid_clock.cc:648] HybridClock initialized: now 1790376582878168 us; error 44 us; skew 500 ppm
Sep 25 22:49:42 dist-test-slave-vdtf krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:42.880851 18174 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:49:42.881750 18174 webserver.cc:460] Webserver started at http://127.0.201.131:43093/ using document root <none> and password file <none>
I20260925 22:49:42.882297 18174 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:42.882382 18174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:42.882601 18174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:42.884619 18174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "ed8b4e699e1b4424982c14efe698a851"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "8ee10aef71f91f0a333cc82e3eecdf8b"
server_key_iv: "2798e1d3e7db5c87a05de98215d53a8d"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.885162 18174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "ed8b4e699e1b4424982c14efe698a851"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "8ee10aef71f91f0a333cc82e3eecdf8b"
server_key_iv: "2798e1d3e7db5c87a05de98215d53a8d"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.888674 18174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:42.890964 18189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.892150 18174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:42.892278 18174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "ed8b4e699e1b4424982c14efe698a851"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-vdtf"
server_key: "8ee10aef71f91f0a333cc82e3eecdf8b"
server_key_iv: "2798e1d3e7db5c87a05de98215d53a8d"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.892387 18174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:42.912292 18174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:42.914992 18174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:42.915198 18174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:42.915777 18174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:42.916568 18174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:42.916640 18174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.916711 18174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:42.916765 18174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.925376 18174 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:42659
I20260925 22:49:42.925401 18302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:42659 every 8 connection(s)
I20260925 22:49:42.926225 18174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:42.927886 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 18174
I20260925 22:49:42.928018 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:42.928313 806 external_mini_cluster.cc:1428] Setting key a4cb20c55bd335201916e20414c6f5a1
Sep 25 22:49:42 dist-test-slave-vdtf krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:42.939391 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.927996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:45817 (local address 127.0.201.190:33365)
0925 22:49:42.928145 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.928149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.929154 (+ 1005us) server_negotiation.cc:407] Connection header received
0925 22:49:42.930461 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.930464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.930517 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.930597 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:42.931723 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.932254 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.933374 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.933606 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.933709 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.936109 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:42.936112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:42.936115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:42.936147 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:42.937438 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.937928 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.937933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.937935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.938019 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.938348 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.938353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.938354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.938533 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:42.938634 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.939112 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.939225 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:49:42.940229 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.928499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:33365 (local address 127.0.201.131:45817)
0925 22:49:42.929006 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:42.929060 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:42.930258 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:42.930762 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:42.930769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:42.931105 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:42.931551 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.931573 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.932379 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.932385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.933239 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.933247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.933753 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.933757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.933915 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.934503 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:42.934519 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:42.935944 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:42.937553 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:42.937559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:42.937580 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:42.937827 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:42.938115 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:42.938120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:42.938122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:42.938255 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:42.938642 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:42.938650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:42.938883 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 22:49:42.939064 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 22:49:42.939371 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":151,"threads_started":1}
I20260925 22:49:42.941342 18303 heartbeater.cc:347] Connected to a master server at 127.0.201.190:33365
I20260925 22:49:42.941581 18303 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:42.942116 18303 heartbeater.cc:511] Master 127.0.201.190:33365 requested a full tablet report, sending...
I20260925 22:49:42.943159 17844 ts_manager.cc:194] Registered new tserver with Master: ed8b4e699e1b4424982c14efe698a851 (127.0.201.131:42659)
I20260925 22:49:42.943797 17844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:45817
I20260925 22:49:42.953722 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:49:42.963142 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.955700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.0.201.190:33365)
0925 22:49:42.955829 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.955834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.956025 (+ 191us) server_negotiation.cc:407] Connection header received
0925 22:49:42.956188 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.956191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.956237 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.956331 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:42.957040 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.957568 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.958466 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.958712 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.958839 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.960241 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:42.960244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:42.960247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:42.960272 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:42.961462 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.961886 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.961890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.961891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.961976 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.962255 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.962258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.962260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.962486 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:42.962628 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.962869 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.962982 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:49:42.966287 17844 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:42.969686 17844 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:42.983695 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.978026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:39503 (local address 127.0.0.1:40714)
0925 22:49:42.978445 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:42.978491 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:42.978619 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:42.979428 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:42.979436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:42.979460 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:42.979603 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.979610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.980849 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.980854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.982036 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.982046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.983152 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.983156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.983260 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.983300 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:49:42.983400 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:49:42.983514 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260925 22:49:42.984015 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.978701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:42659 (local address 127.0.0.1:48200)
0925 22:49:42.978981 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:42.978994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:42.979080 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:42.979804 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:42.979809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:42.979826 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:42.979925 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.979932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.981403 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.981406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.982599 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.982608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.983650 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.983654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.983766 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.983787 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:49:42.983834 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 22:49:42.983886 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":47,"thread_start_us":79,"threads_started":1}
I20260925 22:49:42.984113 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.978170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.0.201.129:39503)
0925 22:49:42.978735 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.978751 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.978776 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:49:42.978850 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.978856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.979061 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.979223 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:42.979750 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.980725 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.982172 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.983021 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.983126 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.983264 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.983625 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.983754 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":91,"threads_started":1}
I20260925 22:49:42.984208 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.979002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.0.201.130:37799)
0925 22:49:42.979541 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.979553 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.979577 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:49:42.979648 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.979656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.979861 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.980015 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:42.980396 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.981468 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.982645 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.983283 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.983401 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.983527 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.983767 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.984059 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":102,"threads_started":1}
I20260925 22:49:42.984427 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.978086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:37799 (local address 127.0.0.1:33150)
0925 22:49:42.978661 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:42.978674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:42.978762 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:42.980170 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:42.980173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:42.980189 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:42.980271 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.980276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.981602 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.981608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.982496 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.982506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.983408 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.983411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.983500 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.983522 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:49:42.983600 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:49:42.983671 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":117,"threads_started":1}
I20260925 22:49:42.984725 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.978956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.0.201.131:42659)
0925 22:49:42.979309 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.979318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.979336 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:49:42.979427 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.979432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.979637 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.980078 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:42.980193 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.981265 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.982731 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.983526 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.984284 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.984422 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.984474 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.984584 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":130,"threads_started":1}
I20260925 22:49:42.986990 18103 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbeea91ee65454dba2252c53642ed5e (DEFAULT_TABLE table=test-table [id=030b7f17b0be47399d9c9270238a7e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:42.987083 17969 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbeea91ee65454dba2252c53642ed5e (DEFAULT_TABLE table=test-table [id=030b7f17b0be47399d9c9270238a7e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:42.987063 18237 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbeea91ee65454dba2252c53642ed5e (DEFAULT_TABLE table=test-table [id=030b7f17b0be47399d9c9270238a7e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:42.988052 18103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbeea91ee65454dba2252c53642ed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:42.988071 17969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbeea91ee65454dba2252c53642ed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:42.988076 18237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbeea91ee65454dba2252c53642ed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:42.994588 18320 tablet_bootstrap.cc:492] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: Bootstrap starting.
I20260925 22:49:42.995309 18321 tablet_bootstrap.cc:492] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8: Bootstrap starting.
I20260925 22:49:42.995309 18322 tablet_bootstrap.cc:492] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: Bootstrap starting.
I20260925 22:49:42.997277 18320 tablet_bootstrap.cc:654] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:42.997417 18321 tablet_bootstrap.cc:654] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:42.997416 18322 tablet_bootstrap.cc:654] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:42.998029 18320 log.cc:824] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:42.998687 18322 log.cc:824] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:42.998687 18321 log.cc:824] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:43.000159 18320 tablet_bootstrap.cc:492] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: No bootstrap required, opened a new log
I20260925 22:49:43.000353 18320 ts_tablet_manager.cc:1397] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260925 22:49:43.000600 18322 tablet_bootstrap.cc:492] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: No bootstrap required, opened a new log
I20260925 22:49:43.000782 18322 ts_tablet_manager.cc:1397] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:49:43.003217 18320 raft_consensus.cc:348] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 } } peers { permanent_uuid: "5c968f1357204a0880373fba0731d906" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 37799 } } peers { permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 } }
I20260925 22:49:43.003456 18320 raft_consensus.cc:374] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:43.003515 18320 raft_consensus.cc:729] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c968f1357204a0880373fba0731d906, State: Initialized, Role: FOLLOWER
I20260925 22:49:43.003549 18322 raft_consensus.cc:348] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 } } peers { permanent_uuid: "5c968f1357204a0880373fba0731d906" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 37799 } } peers { permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 } }
I20260925 22:49:43.003861 18322 raft_consensus.cc:374] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:43.003919 18322 raft_consensus.cc:729] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8b4e699e1b4424982c14efe698a851, State: Initialized, Role: FOLLOWER
I20260925 22:49:43.004005 18320 consensus_queue.cc:260] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 } } peers { permanent_uuid: "5c968f1357204a0880373fba0731d906" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 37799 } } peers { permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 } }
I20260925 22:49:43.004307 18322 consensus_queue.cc:260] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 } } peers { permanent_uuid: "5c968f1357204a0880373fba0731d906" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 37799 } } peers { permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 } }
I20260925 22:49:43.004688 18169 heartbeater.cc:503] Master 127.0.201.190:33365 was elected leader, sending a full tablet report...
I20260925 22:49:43.004874 18303 heartbeater.cc:503] Master 127.0.201.190:33365 was elected leader, sending a full tablet report...
I20260925 22:49:43.005025 18320 ts_tablet_manager.cc:1428] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:43.005218 18322 ts_tablet_manager.cc:1428] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:43.006732 18321 tablet_bootstrap.cc:492] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8: No bootstrap required, opened a new log
I20260925 22:49:43.006944 18321 ts_tablet_manager.cc:1397] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260925 22:49:43.010577 18321 raft_consensus.cc:348] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 } } peers { permanent_uuid: "5c968f1357204a0880373fba0731d906" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 37799 } } peers { permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 } }
I20260925 22:49:43.010802 18321 raft_consensus.cc:374] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:43.010856 18321 raft_consensus.cc:729] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 679a90271eda4aaf821d281c07de2ad8, State: Initialized, Role: FOLLOWER
I20260925 22:49:43.011276 18321 consensus_queue.cc:260] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 } } peers { permanent_uuid: "5c968f1357204a0880373fba0731d906" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 37799 } } peers { permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 } }
I20260925 22:49:43.011969 18035 heartbeater.cc:503] Master 127.0.201.190:33365 was elected leader, sending a full tablet report...
I20260925 22:49:43.012245 18321 ts_tablet_manager.cc:1428] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 22:49:43.177156 18304 tablet.cc:2369] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:49:43.205556 18170 tablet.cc:2369] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:49:43.237528 18036 tablet.cc:2369] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:43.355988 18326 raft_consensus.cc:482] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:43.356251 18326 raft_consensus.cc:504] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 } } peers { permanent_uuid: "5c968f1357204a0880373fba0731d906" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 37799 } } peers { permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 } }
I20260925 22:49:43.357399 18326 leader_election.cc:288] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 679a90271eda4aaf821d281c07de2ad8 (127.0.201.129:39503), ed8b4e699e1b4424982c14efe698a851 (127.0.201.131:42659)
I20260925 22:49:43.361712 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:43.357675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:39503 (local address 127.0.201.130:59399)
0925 22:49:43.357942 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:43.357957 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:43.358084 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:43.358600 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:43.358603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:43.358621 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:43.358772 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:43.358780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.359618 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.359624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:43.360507 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:43.360515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.361246 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.361250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:43.361372 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:43.361404 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 22:49:43.361484 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:49:43.361533 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":58,"thread_start_us":112,"threads_started":1}
I20260925 22:49:43.361718 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:43.357795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:59399 (local address 127.0.201.129:39503)
0925 22:49:43.357937 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:43.357941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:43.358052 (+ 111us) server_negotiation.cc:407] Connection header received
0925 22:49:43.358392 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:43.358397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:43.358464 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:43.358563 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:43.358890 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.359491 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.360633 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.361119 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.361216 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:43.361282 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:43.361516 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 22:49:43.361600 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:49:43.361799 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:43.358254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:49501 (local address 127.0.201.131:42659)
0925 22:49:43.358401 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:43.358404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:43.358418 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:49:43.358579 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:43.358582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:43.358628 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:43.358734 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:43.359063 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.359531 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.360751 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.361203 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.361318 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:43.361375 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:43.361608 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 22:49:43.361672 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:49:43.361768 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:43.358083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:42659 (local address 127.0.201.130:49501)
0925 22:49:43.358308 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:43.358324 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:43.358435 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:43.358767 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:43.358769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:43.358786 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:43.358941 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:43.358949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.359724 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.359730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:43.360629 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:43.360639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.361332 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.361336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:43.361456 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:43.361501 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 22:49:43.361574 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:49:43.361635 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":24,"thread_start_us":74,"threads_started":1}
I20260925 22:49:43.362491 17989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbeea91ee65454dba2252c53642ed5e" candidate_uuid: "5c968f1357204a0880373fba0731d906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "679a90271eda4aaf821d281c07de2ad8" is_pre_election: true
I20260925 22:49:43.362594 18257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbeea91ee65454dba2252c53642ed5e" candidate_uuid: "5c968f1357204a0880373fba0731d906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed8b4e699e1b4424982c14efe698a851" is_pre_election: true
I20260925 22:49:43.362833 17989 raft_consensus.cc:2454] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c968f1357204a0880373fba0731d906 in term 0.
I20260925 22:49:43.362901 18257 raft_consensus.cc:2454] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c968f1357204a0880373fba0731d906 in term 0.
I20260925 22:49:43.363366 18057 leader_election.cc:302] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c968f1357204a0880373fba0731d906, 679a90271eda4aaf821d281c07de2ad8; no voters:
I20260925 22:49:43.363605 18326 raft_consensus.cc:2781] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:43.363721 18326 raft_consensus.cc:482] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:43.363786 18326 raft_consensus.cc:3037] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:43.364915 18326 raft_consensus.cc:504] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 } } peers { permanent_uuid: "5c968f1357204a0880373fba0731d906" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 37799 } } peers { permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 } }
I20260925 22:49:43.365284 18326 leader_election.cc:288] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [CANDIDATE]: Term 1 election: Requested vote from peers 679a90271eda4aaf821d281c07de2ad8 (127.0.201.129:39503), ed8b4e699e1b4424982c14efe698a851 (127.0.201.131:42659)
I20260925 22:49:43.365661 17989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbeea91ee65454dba2252c53642ed5e" candidate_uuid: "5c968f1357204a0880373fba0731d906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "679a90271eda4aaf821d281c07de2ad8"
I20260925 22:49:43.365710 18257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbeea91ee65454dba2252c53642ed5e" candidate_uuid: "5c968f1357204a0880373fba0731d906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed8b4e699e1b4424982c14efe698a851"
I20260925 22:49:43.365795 17989 raft_consensus.cc:3037] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:43.365845 18257 raft_consensus.cc:3037] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:43.367172 17989 raft_consensus.cc:2454] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c968f1357204a0880373fba0731d906 in term 1.
I20260925 22:49:43.367172 18257 raft_consensus.cc:2454] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c968f1357204a0880373fba0731d906 in term 1.
I20260925 22:49:43.367491 18057 leader_election.cc:302] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c968f1357204a0880373fba0731d906, 679a90271eda4aaf821d281c07de2ad8; no voters:
I20260925 22:49:43.367718 18326 raft_consensus.cc:2781] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:43.367949 18326 raft_consensus.cc:686] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 1 LEADER]: Becoming Leader. State: Replica: 5c968f1357204a0880373fba0731d906, State: Running, Role: LEADER
I20260925 22:49:43.368279 18326 consensus_queue.cc:237] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 } } peers { permanent_uuid: "5c968f1357204a0880373fba0731d906" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 37799 } } peers { permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 } }
I20260925 22:49:43.371414 17844 catalog_manager.cc:5495] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c968f1357204a0880373fba0731d906 (127.0.201.130). New cstate: current_term: 1 leader_uuid: "5c968f1357204a0880373fba0731d906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c968f1357204a0880373fba0731d906" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 } health_report { overall_health: UNKNOWN } } }
I20260925 22:49:43.433059 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:43.429786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.0.201.130:37799)
0925 22:49:43.429925 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:43.429929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:43.429943 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:49:43.430028 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:43.430030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:43.430088 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:43.430179 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:43.430401 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.430886 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.431709 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.431917 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.431993 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:43.432247 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:43.432676 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:43.432766 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:43.432816 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 22:49:43.432863 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:49:43.438817 17989 raft_consensus.cc:1261] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5c968f1357204a0880373fba0731d906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:43.438817 18257 raft_consensus.cc:1261] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851 [term 1 FOLLOWER]: Refusing update from remote peer 5c968f1357204a0880373fba0731d906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:43.439556 18331 consensus_queue.cc:1035] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "679a90271eda4aaf821d281c07de2ad8" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:43.439770 18326 consensus_queue.cc:1035] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed8b4e699e1b4424982c14efe698a851" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 42659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:43.448968 18335 mvcc.cc:205] Tried to move back new op lower bound from 7333382485757988864 to 7333382485478969344. Current Snapshot: MvccSnapshot[applied={T|T < 7333382485757988864}]
I20260925 22:49:43.456115 17843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:43.456286 17843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:43.458402 17843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 106aea0b028f4f6cb25d7b8346db94c7: Sending DeleteTablet for 3 replicas of tablet 8dbeea91ee65454dba2252c53642ed5e
I20260925 22:49:43.459115 18103 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbeea91ee65454dba2252c53642ed5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:43 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:33150
I20260925 22:49:43.459192 17969 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbeea91ee65454dba2252c53642ed5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:43 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:40714
I20260925 22:49:43.459300 18237 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbeea91ee65454dba2252c53642ed5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:43 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:48200
I20260925 22:49:43.459631 18345 tablet_replica.cc:330] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8: stopping tablet replica
I20260925 22:49:43.459864 18346 tablet_replica.cc:330] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: stopping tablet replica
I20260925 22:49:43.459955 18345 raft_consensus.cc:2229] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:43.460140 18346 raft_consensus.cc:2229] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:43.460259 18345 raft_consensus.cc:2258] T 8dbeea91ee65454dba2252c53642ed5e P 679a90271eda4aaf821d281c07de2ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:43.460357 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 17906
I20260925 22:49:43.460412 18346 raft_consensus.cc:2258] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:43.462014 18346 ts_tablet_manager.cc:1905] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:43.464432 18344 tablet_replica.cc:330] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: stopping tablet replica
I20260925 22:49:43.464752 18344 raft_consensus.cc:2229] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:43.465257 18344 raft_consensus.cc:2258] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:43.465306 18346 ts_tablet_manager.cc:1918] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:43.465409 18346 log.cc:1196] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/8dbeea91ee65454dba2252c53642ed5e
I20260925 22:49:43.465726 18346 ts_tablet_manager.cc:1939] T 8dbeea91ee65454dba2252c53642ed5e P ed8b4e699e1b4424982c14efe698a851: Deleting consensus metadata
I20260925 22:49:43.466921 17829 catalog_manager.cc:4841] TS ed8b4e699e1b4424982c14efe698a851 (127.0.201.131:42659): tablet 8dbeea91ee65454dba2252c53642ed5e (table test-table [id=030b7f17b0be47399d9c9270238a7e1e]) successfully deleted
I20260925 22:49:43.467263 18344 ts_tablet_manager.cc:1905] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:43.469849 18344 ts_tablet_manager.cc:1918] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:43.469945 18344 log.cc:1196] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/8dbeea91ee65454dba2252c53642ed5e
I20260925 22:49:43.470242 18344 ts_tablet_manager.cc:1939] T 8dbeea91ee65454dba2252c53642ed5e P 5c968f1357204a0880373fba0731d906: Deleting consensus metadata
I20260925 22:49:43.471340 17831 catalog_manager.cc:4841] TS 5c968f1357204a0880373fba0731d906 (127.0.201.130:37799): tablet 8dbeea91ee65454dba2252c53642ed5e (table test-table [id=030b7f17b0be47399d9c9270238a7e1e]) successfully deleted
W20260925 22:49:43.471907 17829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:39503 (error 108)
I20260925 22:49:43.472710 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 18040
I20260925 22:49:43.472924 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:43.472615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:39503 (local address 127.0.0.1:40724)
0925 22:49:43.472739 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:43.472816 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 22:49:43.473400 17829 catalog_manager.cc:4584] TS 679a90271eda4aaf821d281c07de2ad8 (127.0.201.129:39503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8dbeea91ee65454dba2252c53642ed5e: Network error: Client connection negotiation failed: client connection to 127.0.201.129:39503: connect: Connection refused (error 111)
I20260925 22:49:43.478961 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 18174
I20260925 22:49:43.484567 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 17812
2026-09-25T22:49:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2646 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T22:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:43Z Disabled control of system clock
I20260925 22:49:43.513849 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40317
--webserver_interface=127.0.201.190
--webserver_port=0
--builtin_ntp_servers=127.0.201.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:40317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:49:43.621730 18351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:43.621999 18351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:43.622036 18351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:43.626049 18351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:43.626104 18351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:43.626120 18351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:43.626137 18351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:43.626151 18351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:43.630772 18351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:40317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:43.631846 18351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:43.633186 18351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:43.641332 18359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:43.641337 18357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:43.641343 18356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:43.641664 18351 server_base.cc:1034] running on GCE node
I20260925 22:49:43.642202 18351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:43.643219 18351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:43.644414 18351 hybrid_clock.cc:648] HybridClock initialized: now 1790376583644382 us; error 42 us; skew 500 ppm
I20260925 22:49:43.646286 18351 webserver.cc:460] Webserver started at http://127.0.201.190:34049/ using document root <none> and password file <none>
I20260925 22:49:43.646824 18351 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:43.646889 18351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:43.647055 18351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:43.649235 18351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "c429034864e745a3ab802e40e7e14542"
format_stamp: "Formatted at 2026-09-25 22:49:43 on dist-test-slave-vdtf"
server_key: "b9d9545ebc1235308616c93322936cfc"
server_key_iv: "7e99c7705ee690fda06c234f0b884457"
server_key_version: "encryptionkey@0"
I20260925 22:49:43.649746 18351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "c429034864e745a3ab802e40e7e14542"
format_stamp: "Formatted at 2026-09-25 22:49:43 on dist-test-slave-vdtf"
server_key: "b9d9545ebc1235308616c93322936cfc"
server_key_iv: "7e99c7705ee690fda06c234f0b884457"
server_key_version: "encryptionkey@0"
I20260925 22:49:43.653388 18351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 22:49:43.655704 18365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:43.656905 18351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:43.657001 18351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "c429034864e745a3ab802e40e7e14542"
format_stamp: "Formatted at 2026-09-25 22:49:43 on dist-test-slave-vdtf"
server_key: "b9d9545ebc1235308616c93322936cfc"
server_key_iv: "7e99c7705ee690fda06c234f0b884457"
server_key_version: "encryptionkey@0"
I20260925 22:49:43.657094 18351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:43.678429 18351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:43.679040 18351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:43.679241 18351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:43.686039 18351 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:40317
I20260925 22:49:43.686050 18417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:40317 every 8 connection(s)
I20260925 22:49:43.686968 18351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:43.689466 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 18351
I20260925 22:49:43.689592 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:43.689862 806 external_mini_cluster.cc:1428] Setting key 93f37e7496381f1aac3ce31908b946d6
I20260925 22:49:43.690094 18418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:43.697309 18418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542: Bootstrap starting.
I20260925 22:49:43.700091 18418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:43.700342 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:43.691257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.0.201.190:40317)
0925 22:49:43.691940 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:43.691959 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:43.692002 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:49:43.692891 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:43.692916 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:43.693433 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:43.693802 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:43.694265 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.695707 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.696560 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.697230 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.697397 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:43.698263 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:43.698269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:43.698284 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:43.698808 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:43.698879 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:43.698905 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:43.699001 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:43.699100 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 22:49:43.699359 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
I20260925 22:49:43.700925 18418 log.cc:824] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:43.702970 18418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542: No bootstrap required, opened a new log
I20260925 22:49:43.706748 18418 raft_consensus.cc:348] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c429034864e745a3ab802e40e7e14542" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40317 } }
I20260925 22:49:43.706954 18418 raft_consensus.cc:374] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:43.707023 18418 raft_consensus.cc:729] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c429034864e745a3ab802e40e7e14542, State: Initialized, Role: FOLLOWER
I20260925 22:49:43.707445 18418 consensus_queue.cc:260] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c429034864e745a3ab802e40e7e14542" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40317 } }
I20260925 22:49:43.707589 18418 raft_consensus.cc:388] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:43.707652 18418 raft_consensus.cc:482] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:43.707780 18418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:43.708734 18418 raft_consensus.cc:504] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c429034864e745a3ab802e40e7e14542" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40317 } }
I20260925 22:49:43.709061 18418 leader_election.cc:302] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c429034864e745a3ab802e40e7e14542; no voters:
I20260925 22:49:43.709342 18418 leader_election.cc:288] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:43.709467 18423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:43.709705 18423 raft_consensus.cc:686] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [term 1 LEADER]: Becoming Leader. State: Replica: c429034864e745a3ab802e40e7e14542, State: Running, Role: LEADER
I20260925 22:49:43.709992 18423 consensus_queue.cc:237] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c429034864e745a3ab802e40e7e14542" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40317 } }
I20260925 22:49:43.710489 18418 sys_catalog.cc:564] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:43.711699 18424 sys_catalog.cc:455] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c429034864e745a3ab802e40e7e14542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c429034864e745a3ab802e40e7e14542" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40317 } } }
I20260925 22:49:43.711702 18425 sys_catalog.cc:455] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c429034864e745a3ab802e40e7e14542. Latest consensus state: current_term: 1 leader_uuid: "c429034864e745a3ab802e40e7e14542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c429034864e745a3ab802e40e7e14542" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40317 } } }
I20260925 22:49:43.711840 18424 sys_catalog.cc:458] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:43.711874 18425 sys_catalog.cc:458] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:43.712216 18432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:43.715246 18432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:43.720902 18432 catalog_manager.cc:1347] Generated new cluster ID: 3d94f86d4aca4494812d2e1ae32999c1
I20260925 22:49:43.720968 18432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:43.757524 18432 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:43.758833 18432 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:43.770227 18432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542: Generated new TSK 0
I20260925 22:49:43.770924 18432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:43.782747 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40317
--builtin_ntp_servers=127.0.201.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:49:43.900134 18442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:43.900403 18442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:43.900476 18442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:43.904377 18442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:43.904449 18442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:43.904536 18442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:49:43.909183 18442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:43.910171 18442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:43.911522 18442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:43.920848 18450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:43.920848 18448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:43.920890 18447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:43.921023 18442 server_base.cc:1034] running on GCE node
I20260925 22:49:43.921576 18442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:43.922160 18442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:43.923318 18442 hybrid_clock.cc:648] HybridClock initialized: now 1790376583923299 us; error 28 us; skew 500 ppm
I20260925 22:49:43.925247 18442 webserver.cc:460] Webserver started at http://127.0.201.129:32785/ using document root <none> and password file <none>
I20260925 22:49:43.925830 18442 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:43.925891 18442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:43.926139 18442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:43.928215 18442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "a0837b2c331347f6947491824a7ec760"
format_stamp: "Formatted at 2026-09-25 22:49:43 on dist-test-slave-vdtf"
server_key: "3d7a21dd970b87993d87be3d14560e92"
server_key_iv: "be74396af106a440c7c0020faf9e03c9"
server_key_version: "encryptionkey@0"
I20260925 22:49:43.928771 18442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "a0837b2c331347f6947491824a7ec760"
format_stamp: "Formatted at 2026-09-25 22:49:43 on dist-test-slave-vdtf"
server_key: "3d7a21dd970b87993d87be3d14560e92"
server_key_iv: "be74396af106a440c7c0020faf9e03c9"
server_key_version: "encryptionkey@0"
I20260925 22:49:43.932580 18442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:49:43.934907 18456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:43.936156 18442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:43.936298 18442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "a0837b2c331347f6947491824a7ec760"
format_stamp: "Formatted at 2026-09-25 22:49:43 on dist-test-slave-vdtf"
server_key: "3d7a21dd970b87993d87be3d14560e92"
server_key_iv: "be74396af106a440c7c0020faf9e03c9"
server_key_version: "encryptionkey@0"
I20260925 22:49:43.936415 18442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:43.991392 18442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:43.992241 18442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:43.992453 18442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:43.993059 18442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:43.994019 18442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:43.994091 18442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:43.994164 18442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:43.994210 18442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:44.002794 18442 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:35219
I20260925 22:49:44.002823 18569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:35219 every 8 connection(s)
I20260925 22:49:44.003718 18442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:44.010134 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 18442
I20260925 22:49:44.010269 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:44.010553 806 external_mini_cluster.cc:1428] Setting key 17500bf7bd21adb317ad94173e7c24b8
I20260925 22:49:44.012665 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40317
--builtin_ntp_servers=127.0.201.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:49:44.013111 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.005544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:38713 (local address 127.0.201.190:40317)
0925 22:49:44.005680 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.005684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.006719 (+ 1035us) server_negotiation.cc:407] Connection header received
0925 22:49:44.008120 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.008122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.008171 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.008247 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:44.009124 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.009648 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.010972 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.011304 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.011412 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.012642 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:44.012645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:44.012647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:44.012658 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:44.012678 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:44.012686 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:44.012766 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.012924 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.012999 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:49:44.014236 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.005921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40317 (local address 127.0.201.129:38713)
0925 22:49:44.006558 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:44.006597 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:44.007916 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:44.008410 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:44.008417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:44.008500 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:44.008991 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.009003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.009748 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.009751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.010843 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.010852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.011465 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.011468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.011616 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.012353 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:44.012357 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:44.012413 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:44.012496 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:44.012775 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:44.012786 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:44.012794 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:49:44.012920 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 22:49:44.013224 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
I20260925 22:49:44.015367 18570 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40317
I20260925 22:49:44.015617 18570 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:44.016148 18570 heartbeater.cc:511] Master 127.0.201.190:40317 requested a full tablet report, sending...
I20260925 22:49:44.017797 18382 ts_manager.cc:194] Registered new tserver with Master: a0837b2c331347f6947491824a7ec760 (127.0.201.129:35219)
I20260925 22:49:44.019564 18382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.129:38713
W20260925 22:49:44.126746 18573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:44.126997 18573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:44.127053 18573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:44.131089 18573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:44.131146 18573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:44.131201 18573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:49:44.135814 18573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:44.136802 18573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:44.138132 18573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:44.147953 18578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:44.148082 18573 server_base.cc:1034] running on GCE node
W20260925 22:49:44.148001 18581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:44.147943 18579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:44.148623 18573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:44.149240 18573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:44.150436 18573 hybrid_clock.cc:648] HybridClock initialized: now 1790376584150419 us; error 26 us; skew 500 ppm
I20260925 22:49:44.152329 18573 webserver.cc:460] Webserver started at http://127.0.201.130:33841/ using document root <none> and password file <none>
I20260925 22:49:44.152959 18573 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:44.153051 18573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:44.153301 18573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:44.155488 18573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "9d3d231d19e242b9b979c9e78aa37da0"
format_stamp: "Formatted at 2026-09-25 22:49:44 on dist-test-slave-vdtf"
server_key: "3237fe0e3e93ab840339fa4887ab571f"
server_key_iv: "c3e0c715a1083a72b5edb629acb7f75b"
server_key_version: "encryptionkey@0"
I20260925 22:49:44.156111 18573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "9d3d231d19e242b9b979c9e78aa37da0"
format_stamp: "Formatted at 2026-09-25 22:49:44 on dist-test-slave-vdtf"
server_key: "3237fe0e3e93ab840339fa4887ab571f"
server_key_iv: "c3e0c715a1083a72b5edb629acb7f75b"
server_key_version: "encryptionkey@0"
I20260925 22:49:44.160100 18573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:49:44.162572 18587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:44.163895 18573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:44.164026 18573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "9d3d231d19e242b9b979c9e78aa37da0"
format_stamp: "Formatted at 2026-09-25 22:49:44 on dist-test-slave-vdtf"
server_key: "3237fe0e3e93ab840339fa4887ab571f"
server_key_iv: "c3e0c715a1083a72b5edb629acb7f75b"
server_key_version: "encryptionkey@0"
I20260925 22:49:44.164147 18573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:44.178973 18573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:44.179658 18573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:44.179898 18573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:44.180471 18573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:44.181469 18573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:44.181529 18573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:44.181578 18573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:44.181612 18573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:44.189846 18573 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:43549
I20260925 22:49:44.189872 18700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:43549 every 8 connection(s)
I20260925 22:49:44.190713 18573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:44.199424 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 18573
I20260925 22:49:44.199544 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:44.199641 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.192495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:52457 (local address 127.0.201.190:40317)
0925 22:49:44.192630 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.192634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.193665 (+ 1031us) server_negotiation.cc:407] Connection header received
0925 22:49:44.195042 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.195045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.195105 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.195185 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:44.196072 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.196610 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.197789 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.198030 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.198143 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.199132 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:44.199135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:44.199138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:44.199152 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:44.199179 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:44.199187 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:44.199263 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.199429 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.199497 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:44.199880 806 external_mini_cluster.cc:1428] Setting key 181dd42414b981ae2913d062ad817d35
I20260925 22:49:44.200634 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.192880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40317 (local address 127.0.201.130:52457)
0925 22:49:44.193502 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:44.193540 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:44.194832 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:44.195347 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:44.195353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:44.195441 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:44.195926 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.195938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.196734 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.196739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.197661 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.197669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.198179 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.198182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.198346 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.198879 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:44.198880 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:44.198919 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:44.198982 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:44.199276 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:44.199286 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:44.199294 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:49:44.199422 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 22:49:44.199737 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260925 22:49:44.201821 18701 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40317
I20260925 22:49:44.201821 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40317
--builtin_ntp_servers=127.0.201.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:49:44.202049 18701 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:44.202636 18701 heartbeater.cc:511] Master 127.0.201.190:40317 requested a full tablet report, sending...
I20260925 22:49:44.203748 18382 ts_manager.cc:194] Registered new tserver with Master: 9d3d231d19e242b9b979c9e78aa37da0 (127.0.201.130:43549)
I20260925 22:49:44.204552 18382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.130:52457
W20260925 22:49:44.311143 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:44.311452 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:44.311530 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:44.315618 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:44.315713 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:44.315824 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:49:44.320483 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:44.321489 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:44.322798 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:44.331969 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:44.331990 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:44.331990 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:44.332037 18704 server_base.cc:1034] running on GCE node
I20260925 22:49:44.332607 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:44.333202 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:44.334388 18704 hybrid_clock.cc:648] HybridClock initialized: now 1790376584334365 us; error 39 us; skew 500 ppm
I20260925 22:49:44.336131 18704 webserver.cc:460] Webserver started at http://127.0.201.131:43115/ using document root <none> and password file <none>
I20260925 22:49:44.336696 18704 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:44.336783 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:44.337013 18704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:44.338975 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "c1e20d74d629477aa25d341c5009544d"
format_stamp: "Formatted at 2026-09-25 22:49:44 on dist-test-slave-vdtf"
server_key: "64ed368401826ec68361d8ce54621392"
server_key_iv: "3c3d3692c0133767c2916b9360b19669"
server_key_version: "encryptionkey@0"
I20260925 22:49:44.339550 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "c1e20d74d629477aa25d341c5009544d"
format_stamp: "Formatted at 2026-09-25 22:49:44 on dist-test-slave-vdtf"
server_key: "64ed368401826ec68361d8ce54621392"
server_key_iv: "3c3d3692c0133767c2916b9360b19669"
server_key_version: "encryptionkey@0"
I20260925 22:49:44.343408 18704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 22:49:44.345892 18718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:44.347110 18704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:44.347241 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "c1e20d74d629477aa25d341c5009544d"
format_stamp: "Formatted at 2026-09-25 22:49:44 on dist-test-slave-vdtf"
server_key: "64ed368401826ec68361d8ce54621392"
server_key_iv: "3c3d3692c0133767c2916b9360b19669"
server_key_version: "encryptionkey@0"
I20260925 22:49:44.347352 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:44.364471 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:44.365152 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:44.365341 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:44.365906 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:44.366799 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:44.366869 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:44.366986 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:44.367028 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:44.375514 18704 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:37583
I20260925 22:49:44.375536 18831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:37583 every 8 connection(s)
I20260925 22:49:44.376415 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:44.378352 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 18704
I20260925 22:49:44.378472 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:44.378762 806 external_mini_cluster.cc:1428] Setting key 4ec71cae2ba844eca94bf2e47e4839b8
I20260925 22:49:44.385206 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.378208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:41563 (local address 127.0.201.190:40317)
0925 22:49:44.378360 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.378365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.379492 (+ 1127us) server_negotiation.cc:407] Connection header received
0925 22:49:44.381078 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.381081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.381125 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.381195 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:44.382027 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.382524 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.383590 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.383809 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.383882 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.384830 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:44.384832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:44.384835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:44.384845 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:44.384864 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:44.384871 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:44.384926 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.385052 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.385101 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:49:44.385965 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.378632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40317 (local address 127.0.201.131:41563)
0925 22:49:44.379308 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:44.379370 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:44.380902 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:44.381354 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:44.381360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:44.381452 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:44.381915 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.381927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.382633 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.382635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.383493 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.383501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.383937 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.383940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.384082 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.384608 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:44.384609 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:44.384647 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:44.384705 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:44.384940 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:44.384946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:44.384949 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:49:44.385038 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:49:44.385235 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":134,"threads_started":1}
I20260925 22:49:44.386915 18832 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40317
I20260925 22:49:44.387137 18832 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:44.387599 18832 heartbeater.cc:511] Master 127.0.201.190:40317 requested a full tablet report, sending...
I20260925 22:49:44.388639 18382 ts_manager.cc:194] Registered new tserver with Master: c1e20d74d629477aa25d341c5009544d (127.0.201.131:37583)
I20260925 22:49:44.389171 18382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.131:41563
I20260925 22:49:44.392395 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:49:44.397518 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.394248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.0.201.190:40317)
0925 22:49:44.394359 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.394362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.394570 (+ 208us) server_negotiation.cc:407] Connection header received
0925 22:49:44.394656 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.394658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.394692 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.394756 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:44.395000 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.395465 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.396297 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.396493 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.396576 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.397228 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:44.397231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:44.397233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:44.397243 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:44.397258 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:44.397266 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:44.397319 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.397389 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.397428 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:49:44.399932 18382 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:44.402720 18382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:44.417181 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.411124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.0.201.131:37583)
0925 22:49:44.411894 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.411905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.411926 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:49:44.412015 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.412022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.412231 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.412829 (+ 598us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:44.412982 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.414140 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.415943 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.416500 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.416623 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.416814 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.416929 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.417021 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":177,"threads_started":1}
I20260925 22:49:44.417176 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.410964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:37583 (local address 127.0.0.1:37462)
0925 22:49:44.411319 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:44.411372 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:44.411497 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:44.412433 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:44.412440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:44.412467 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:44.412635 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.412643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.414268 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.414271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.415805 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.415816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.416625 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.416628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.416741 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.416778 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:49:44.416901 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:49:44.417021 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":81,"thread_start_us":84,"threads_started":1}
I20260925 22:49:44.417420 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.411735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:43549 (local address 127.0.0.1:43170)
0925 22:49:44.412812 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:44.412824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:44.412911 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:44.413339 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:44.413344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:44.413364 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:44.413439 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.413445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.414672 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.414675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.415644 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.415653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.416891 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.416894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.416980 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.417002 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:49:44.417076 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:49:44.417151 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":90,"threads_started":1}
I20260925 22:49:44.417761 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.412155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.0.201.130:43549)
0925 22:49:44.412596 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.412608 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.412920 (+ 312us) server_negotiation.cc:407] Connection header received
0925 22:49:44.413003 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.413019 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.413192 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.413335 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:44.413532 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.414545 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.415800 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.416540 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.416649 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.416826 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.417388 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.417570 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":116,"threads_started":1}
I20260925 22:49:44.417914 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.411798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.0.201.129:35219)
0925 22:49:44.412195 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.412210 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.412615 (+ 405us) server_negotiation.cc:407] Connection header received
0925 22:49:44.412750 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.412756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.412951 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.413115 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:44.413308 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.414120 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.416048 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.416793 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.416896 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.417077 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.417636 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.417735 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":147,"threads_started":1}
I20260925 22:49:44.418481 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.411174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:35219 (local address 127.0.0.1:45828)
0925 22:49:44.412504 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:44.412518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:44.412639 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:44.413094 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:44.413096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:44.413113 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:44.413212 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.413221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.414241 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.414244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.415930 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.415938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.417131 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.417135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.417216 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.417237 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:49:44.418262 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0925 22:49:44.418352 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":116,"thread_start_us":85,"threads_started":1}
I20260925 22:49:44.420459 18635 tablet_service.cc:1467] Processing CreateTablet for tablet b6b3bfd2c9d349df923d87b166fdd47b (DEFAULT_TABLE table=test-table [id=9456dc155f994e8e81212ca3ab4b83c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:44.420352 18766 tablet_service.cc:1467] Processing CreateTablet for tablet b6b3bfd2c9d349df923d87b166fdd47b (DEFAULT_TABLE table=test-table [id=9456dc155f994e8e81212ca3ab4b83c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:44.421358 18635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b3bfd2c9d349df923d87b166fdd47b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:44.421008 18504 tablet_service.cc:1467] Processing CreateTablet for tablet b6b3bfd2c9d349df923d87b166fdd47b (DEFAULT_TABLE table=test-table [id=9456dc155f994e8e81212ca3ab4b83c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:44.421383 18766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b3bfd2c9d349df923d87b166fdd47b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:44.421871 18504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b3bfd2c9d349df923d87b166fdd47b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:44.427436 18849 tablet_bootstrap.cc:492] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: Bootstrap starting.
I20260925 22:49:44.428821 18850 tablet_bootstrap.cc:492] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: Bootstrap starting.
I20260925 22:49:44.429325 18851 tablet_bootstrap.cc:492] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760: Bootstrap starting.
I20260925 22:49:44.429870 18849 tablet_bootstrap.cc:654] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:44.430599 18849 log.cc:824] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:44.431031 18850 tablet_bootstrap.cc:654] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:44.431461 18851 tablet_bootstrap.cc:654] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:44.431870 18850 log.cc:824] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:44.432322 18851 log.cc:824] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:44.433199 18849 tablet_bootstrap.cc:492] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: No bootstrap required, opened a new log
I20260925 22:49:44.433398 18849 ts_tablet_manager.cc:1397] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:49:44.434067 18851 tablet_bootstrap.cc:492] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760: No bootstrap required, opened a new log
I20260925 22:49:44.434254 18851 ts_tablet_manager.cc:1397] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:49:44.436666 18849 raft_consensus.cc:348] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e20d74d629477aa25d341c5009544d" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37583 } } peers { permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 } } peers { permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 } }
I20260925 22:49:44.436901 18849 raft_consensus.cc:374] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:44.436959 18849 raft_consensus.cc:729] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3d231d19e242b9b979c9e78aa37da0, State: Initialized, Role: FOLLOWER
I20260925 22:49:44.437424 18849 consensus_queue.cc:260] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e20d74d629477aa25d341c5009544d" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37583 } } peers { permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 } } peers { permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 } }
I20260925 22:49:44.437424 18851 raft_consensus.cc:348] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e20d74d629477aa25d341c5009544d" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37583 } } peers { permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 } } peers { permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 } }
I20260925 22:49:44.437639 18851 raft_consensus.cc:374] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:44.437696 18851 raft_consensus.cc:729] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0837b2c331347f6947491824a7ec760, State: Initialized, Role: FOLLOWER
I20260925 22:49:44.438123 18701 heartbeater.cc:503] Master 127.0.201.190:40317 was elected leader, sending a full tablet report...
I20260925 22:49:44.438405 18849 ts_tablet_manager.cc:1428] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:44.438581 18850 tablet_bootstrap.cc:492] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: No bootstrap required, opened a new log
I20260925 22:49:44.438896 18850 ts_tablet_manager.cc:1397] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260925 22:49:44.439219 18851 consensus_queue.cc:260] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e20d74d629477aa25d341c5009544d" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37583 } } peers { permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 } } peers { permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 } }
I20260925 22:49:44.440105 18570 heartbeater.cc:503] Master 127.0.201.190:40317 was elected leader, sending a full tablet report...
I20260925 22:49:44.440212 18851 ts_tablet_manager.cc:1428] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 22:49:44.441602 18702 tablet.cc:2369] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:44.443148 18850 raft_consensus.cc:348] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e20d74d629477aa25d341c5009544d" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37583 } } peers { permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 } } peers { permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 } }
I20260925 22:49:44.443418 18850 raft_consensus.cc:374] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:44.443486 18850 raft_consensus.cc:729] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e20d74d629477aa25d341c5009544d, State: Initialized, Role: FOLLOWER
I20260925 22:49:44.443897 18850 consensus_queue.cc:260] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e20d74d629477aa25d341c5009544d" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37583 } } peers { permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 } } peers { permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 } }
I20260925 22:49:44.444489 18832 heartbeater.cc:503] Master 127.0.201.190:40317 was elected leader, sending a full tablet report...
I20260925 22:49:44.444784 18850 ts_tablet_manager.cc:1428] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260925 22:49:44.504992 18571 tablet.cc:2369] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:44.567531 18857 raft_consensus.cc:482] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:44.567809 18857 raft_consensus.cc:504] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e20d74d629477aa25d341c5009544d" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37583 } } peers { permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 } } peers { permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 } }
I20260925 22:49:44.569062 18857 leader_election.cc:288] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d3d231d19e242b9b979c9e78aa37da0 (127.0.201.130:43549), a0837b2c331347f6947491824a7ec760 (127.0.201.129:35219)
I20260925 22:49:44.572558 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.569514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:50031 (local address 127.0.201.129:35219)
0925 22:49:44.569646 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.569650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.569672 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:49:44.569719 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.569721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.569760 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.569852 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:44.570123 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.570614 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.571626 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.572023 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.572150 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.572201 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.572375 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.572440 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:44.572558 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.569287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:35219 (local address 127.0.201.131:50031)
0925 22:49:44.569435 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:44.569452 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:44.569556 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:44.569864 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:44.569867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:44.569884 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:44.570000 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.570007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.570727 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.570732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.571492 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.571509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.572144 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.572148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.572253 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.572271 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:49:44.572361 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:49:44.572430 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 22:49:44.572979 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.569359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:60003 (local address 127.0.201.130:43549)
0925 22:49:44.569497 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.569504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.569796 (+ 292us) server_negotiation.cc:407] Connection header received
0925 22:49:44.569950 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.569953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.570003 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.570095 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:44.570359 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.570837 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.571906 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.572305 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.572435 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.572480 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.572806 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.572881 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:49:44.573300 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.569289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:43549 (local address 127.0.201.131:60003)
0925 22:49:44.569697 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:44.569708 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:44.569835 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:44.570106 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:44.570109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:44.570123 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:44.570251 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.570256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.570948 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.570953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.571791 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:44.571800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.572435 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.572439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:44.572533 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.572554 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:49:44.573126 (+ 572us) client_negotiation.cc:240] Negotiation successful
0925 22:49:44.573202 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":58,"thread_start_us":125,"threads_started":1}
I20260925 22:49:44.573257 18524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b3bfd2c9d349df923d87b166fdd47b" candidate_uuid: "c1e20d74d629477aa25d341c5009544d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0837b2c331347f6947491824a7ec760" is_pre_election: true
I20260925 22:49:44.573580 18524 raft_consensus.cc:2454] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e20d74d629477aa25d341c5009544d in term 0.
I20260925 22:49:44.573899 18655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b3bfd2c9d349df923d87b166fdd47b" candidate_uuid: "c1e20d74d629477aa25d341c5009544d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3d231d19e242b9b979c9e78aa37da0" is_pre_election: true
I20260925 22:49:44.574111 18720 leader_election.cc:302] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0837b2c331347f6947491824a7ec760, c1e20d74d629477aa25d341c5009544d; no voters:
I20260925 22:49:44.574225 18655 raft_consensus.cc:2454] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e20d74d629477aa25d341c5009544d in term 0.
I20260925 22:49:44.574379 18857 raft_consensus.cc:2781] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:44.574491 18857 raft_consensus.cc:482] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:44.574555 18857 raft_consensus.cc:3037] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:44.575527 18857 raft_consensus.cc:504] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e20d74d629477aa25d341c5009544d" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37583 } } peers { permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 } } peers { permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 } }
I20260925 22:49:44.575905 18857 leader_election.cc:288] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [CANDIDATE]: Term 1 election: Requested vote from peers 9d3d231d19e242b9b979c9e78aa37da0 (127.0.201.130:43549), a0837b2c331347f6947491824a7ec760 (127.0.201.129:35219)
I20260925 22:49:44.576264 18655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b3bfd2c9d349df923d87b166fdd47b" candidate_uuid: "c1e20d74d629477aa25d341c5009544d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3d231d19e242b9b979c9e78aa37da0"
I20260925 22:49:44.576409 18655 raft_consensus.cc:3037] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:44.576429 18524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b3bfd2c9d349df923d87b166fdd47b" candidate_uuid: "c1e20d74d629477aa25d341c5009544d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0837b2c331347f6947491824a7ec760"
I20260925 22:49:44.576548 18524 raft_consensus.cc:3037] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:44.577729 18655 raft_consensus.cc:2454] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e20d74d629477aa25d341c5009544d in term 1.
I20260925 22:49:44.577729 18524 raft_consensus.cc:2454] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e20d74d629477aa25d341c5009544d in term 1.
I20260925 22:49:44.578054 18722 leader_election.cc:302] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d3d231d19e242b9b979c9e78aa37da0, c1e20d74d629477aa25d341c5009544d; no voters:
I20260925 22:49:44.578256 18857 raft_consensus.cc:2781] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:44.578438 18857 raft_consensus.cc:686] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 1 LEADER]: Becoming Leader. State: Replica: c1e20d74d629477aa25d341c5009544d, State: Running, Role: LEADER
I20260925 22:49:44.578751 18857 consensus_queue.cc:237] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e20d74d629477aa25d341c5009544d" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37583 } } peers { permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 } } peers { permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 } }
I20260925 22:49:44.581996 18382 catalog_manager.cc:5495] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d reported cstate change: term changed from 0 to 1, leader changed from <none> to c1e20d74d629477aa25d341c5009544d (127.0.201.131). New cstate: current_term: 1 leader_uuid: "c1e20d74d629477aa25d341c5009544d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e20d74d629477aa25d341c5009544d" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 } health_report { overall_health: UNKNOWN } } }
I20260925 22:49:44.603861 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.600712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.0.201.131:37583)
0925 22:49:44.600840 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.600844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.600856 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:49:44.600925 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.600927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.600979 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.601068 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:44.601272 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.601754 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.602545 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.602745 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.602817 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.603068 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:44.603462 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:44.603548 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.603602 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.603645 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:49:44.609475 18655 raft_consensus.cc:1261] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0 [term 1 FOLLOWER]: Refusing update from remote peer c1e20d74d629477aa25d341c5009544d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:44.609474 18524 raft_consensus.cc:1261] T b6b3bfd2c9d349df923d87b166fdd47b P a0837b2c331347f6947491824a7ec760 [term 1 FOLLOWER]: Refusing update from remote peer c1e20d74d629477aa25d341c5009544d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:44.610193 18859 consensus_queue.cc:1035] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0837b2c331347f6947491824a7ec760" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:44.610354 18857 consensus_queue.cc:1035] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d3d231d19e242b9b979c9e78aa37da0" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:44.617663 18864 mvcc.cc:205] Tried to move back new op lower bound from 7333382490553344000 to 7333382490437398528. Current Snapshot: MvccSnapshot[applied={T|T < 7333382490553344000}]
I20260925 22:49:44.618810 18862 mvcc.cc:205] Tried to move back new op lower bound from 7333382490553344000 to 7333382490437398528. Current Snapshot: MvccSnapshot[applied={T|T < 7333382490553344000 or (T in {7333382490553344000})}]
I20260925 22:49:44.624598 18382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:44.624758 18382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:44.626880 18382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c429034864e745a3ab802e40e7e14542: Sending DeleteTablet for 3 replicas of tablet b6b3bfd2c9d349df923d87b166fdd47b
I20260925 22:49:44.627763 18766 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b3bfd2c9d349df923d87b166fdd47b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:44 UTC) from {username='slave'} at 127.0.0.1:37462
I20260925 22:49:44.627761 18504 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b3bfd2c9d349df923d87b166fdd47b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:44 UTC) from {username='slave'} at 127.0.0.1:45828
I20260925 22:49:44.628026 18635 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b3bfd2c9d349df923d87b166fdd47b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:44 UTC) from {username='slave'} at 127.0.0.1:43170
I20260925 22:49:44.628324 18872 tablet_replica.cc:330] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: stopping tablet replica
I20260925 22:49:44.628494 18874 tablet_replica.cc:330] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: stopping tablet replica
I20260925 22:49:44.628600 18872 raft_consensus.cc:2229] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:44.628775 18874 raft_consensus.cc:2229] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:44.628927 18872 raft_consensus.cc:2258] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:44.629087 18874 raft_consensus.cc:2258] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:44.629145 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 18442
I20260925 22:49:44.631428 18872 ts_tablet_manager.cc:1905] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:44.633260 18874 ts_tablet_manager.cc:1905] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:44.634205 18872 ts_tablet_manager.cc:1918] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:44.634300 18872 log.cc:1196] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/b6b3bfd2c9d349df923d87b166fdd47b
I20260925 22:49:44.634617 18872 ts_tablet_manager.cc:1939] T b6b3bfd2c9d349df923d87b166fdd47b P c1e20d74d629477aa25d341c5009544d: Deleting consensus metadata
I20260925 22:49:44.635779 18369 catalog_manager.cc:4841] TS c1e20d74d629477aa25d341c5009544d (127.0.201.131:37583): tablet b6b3bfd2c9d349df923d87b166fdd47b (table test-table [id=9456dc155f994e8e81212ca3ab4b83c2]) successfully deleted
I20260925 22:49:44.635968 18874 ts_tablet_manager.cc:1918] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:44.636073 18874 log.cc:1196] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/b6b3bfd2c9d349df923d87b166fdd47b
I20260925 22:49:44.636354 18874 ts_tablet_manager.cc:1939] T b6b3bfd2c9d349df923d87b166fdd47b P 9d3d231d19e242b9b979c9e78aa37da0: Deleting consensus metadata
I20260925 22:49:44.637372 18369 catalog_manager.cc:4841] TS 9d3d231d19e242b9b979c9e78aa37da0 (127.0.201.130:43549): tablet b6b3bfd2c9d349df923d87b166fdd47b (table test-table [id=9456dc155f994e8e81212ca3ab4b83c2]) successfully deleted
W20260925 22:49:44.638335 18367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:35219 (error 108)
I20260925 22:49:44.638554 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 18573
I20260925 22:49:44.639600 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.639279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:35219 (local address 127.0.0.1:45840)
0925 22:49:44.639433 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:44.639517 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 22:49:44.639784 18367 catalog_manager.cc:4584] TS a0837b2c331347f6947491824a7ec760 (127.0.201.129:35219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6b3bfd2c9d349df923d87b166fdd47b: Network error: Client connection negotiation failed: client connection to 127.0.201.129:35219: connect: Connection refused (error 111)
I20260925 22:49:44.645598 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 18704
I20260925 22:49:44.651024 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 18351
2026-09-25T22:49:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1166 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:44 dist-test-slave-vdtf krb5kdc[18878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:44 dist-test-slave-vdtf krb5kdc[18878](info): set up 2 sockets
Sep 25 22:49:44 dist-test-slave-vdtf krb5kdc[18878](info): commencing operation
krb5kdc: starting...
W20260925 22:49:45.707458 806 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:49:45 dist-test-slave-vdtf krb5kdc[18878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:49:45.827304 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40737
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:40737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:45.937722 18891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:45.937979 18891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:45.938019 18891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:45.941865 18891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:45.941918 18891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:45.941937 18891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:45.941954 18891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:45.941969 18891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:45.947016 18891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:40737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:45.948037 18891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:45.948988 18891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:45.957494 18896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:45.957479 18897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:45.957494 18899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:45.957546 18891 server_base.cc:1034] running on GCE node
I20260925 22:49:45.958210 18891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:45.959229 18891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:45.960381 18891 hybrid_clock.cc:648] HybridClock initialized: now 1790376585960362 us; error 31 us; skew 500 ppm
Sep 25 22:49:45 dist-test-slave-vdtf krb5kdc[18878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:45.967433 18891 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:49:45.968799 18891 webserver.cc:460] Webserver started at http://127.0.201.190:34083/ using document root <none> and password file <none>
I20260925 22:49:45.969473 18891 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:45.969574 18891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:45.969830 18891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:45.972193 18891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "69b0f7fc81f44af5ad4bd4640704bf66"
format_stamp: "Formatted at 2026-09-25 22:49:45 on dist-test-slave-vdtf"
server_key: "970638300761b0036dcd58d48090f174"
server_key_iv: "c106ee03070ae7b13aa5d89a5152b7fe"
server_key_version: "encryptionkey@0"
I20260925 22:49:45.972961 18891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "69b0f7fc81f44af5ad4bd4640704bf66"
format_stamp: "Formatted at 2026-09-25 22:49:45 on dist-test-slave-vdtf"
server_key: "970638300761b0036dcd58d48090f174"
server_key_iv: "c106ee03070ae7b13aa5d89a5152b7fe"
server_key_version: "encryptionkey@0"
I20260925 22:49:45.978202 18891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:45.981899 18906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:45.983781 18891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:45.984009 18891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "69b0f7fc81f44af5ad4bd4640704bf66"
format_stamp: "Formatted at 2026-09-25 22:49:45 on dist-test-slave-vdtf"
server_key: "970638300761b0036dcd58d48090f174"
server_key_iv: "c106ee03070ae7b13aa5d89a5152b7fe"
server_key_version: "encryptionkey@0"
I20260925 22:49:45.984251 18891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:45.997965 18891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:46.002422 18891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:46.002646 18891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:46.011849 18891 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:40737
I20260925 22:49:46.011876 18958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:40737 every 8 connection(s)
I20260925 22:49:46.012845 18891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:46.013564 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 18891
I20260925 22:49:46.013788 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:46.014377 806 external_mini_cluster.cc:1428] Setting key bd2c121a2d4b9a2947e772feaabadb5e
I20260925 22:49:46.019120 18959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 22:49:46 dist-test-slave-vdtf krb5kdc[18878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:46.025794 18959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66: Bootstrap starting.
I20260925 22:49:46.028394 18959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:46.029304 18959 log.cc:824] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:46.030007 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.016261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.0.201.190:40737)
0925 22:49:46.016731 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.016773 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.016817 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:49:46.017907 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.017952 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.018474 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.018899 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:46.020029 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.021059 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.021924 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.022411 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.022593 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.025145 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:46.025151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:46.025163 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:46.025189 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:46.026927 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.027456 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.027463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.027470 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.027631 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.027978 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.027989 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.027991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.028367 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:46.028567 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.028886 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.029173 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":298,"thread_start_us":129,"threads_started":1}
I20260925 22:49:46.031531 18959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66: No bootstrap required, opened a new log
I20260925 22:49:46.035351 18959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b0f7fc81f44af5ad4bd4640704bf66" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40737 } }
I20260925 22:49:46.035567 18959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:46.035657 18959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b0f7fc81f44af5ad4bd4640704bf66, State: Initialized, Role: FOLLOWER
I20260925 22:49:46.036108 18959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b0f7fc81f44af5ad4bd4640704bf66" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40737 } }
I20260925 22:49:46.036254 18959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:46.036324 18959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:46.036450 18959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:46.037353 18959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b0f7fc81f44af5ad4bd4640704bf66" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40737 } }
I20260925 22:49:46.037716 18959 leader_election.cc:302] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69b0f7fc81f44af5ad4bd4640704bf66; no voters:
I20260925 22:49:46.038096 18959 leader_election.cc:288] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:46.038237 18964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:46.038507 18964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [term 1 LEADER]: Becoming Leader. State: Replica: 69b0f7fc81f44af5ad4bd4640704bf66, State: Running, Role: LEADER
I20260925 22:49:46.038829 18964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b0f7fc81f44af5ad4bd4640704bf66" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40737 } }
I20260925 22:49:46.039361 18959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:46.040580 18966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69b0f7fc81f44af5ad4bd4640704bf66. Latest consensus state: current_term: 1 leader_uuid: "69b0f7fc81f44af5ad4bd4640704bf66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b0f7fc81f44af5ad4bd4640704bf66" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40737 } } }
I20260925 22:49:46.040628 18965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69b0f7fc81f44af5ad4bd4640704bf66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b0f7fc81f44af5ad4bd4640704bf66" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40737 } } }
I20260925 22:49:46.040694 18966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:46.040746 18965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:46.041101 18969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:46.044595 18969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:46.050012 18969 catalog_manager.cc:1347] Generated new cluster ID: b73936add31746d1af3779451a9d95a7
I20260925 22:49:46.050076 18969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:46.068621 18969 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:46.069904 18969 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:46.079898 18969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66: Generated new TSK 0
I20260925 22:49:46.080713 18969 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:49:46.115600 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40737
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:46.226482 18985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:46.226737 18985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:46.226778 18985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:46.230932 18985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:46.230994 18985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:46.231093 18985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:49:46.236397 18985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:46.237429 18985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:46.238449 18985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:46.248185 18991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:46.248170 18993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:46.248178 18990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.248565 18985 server_base.cc:1034] running on GCE node
I20260925 22:49:46.248994 18985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:46.249617 18985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:46.250914 18985 hybrid_clock.cc:648] HybridClock initialized: now 1790376586250896 us; error 30 us; skew 500 ppm
Sep 25 22:49:46 dist-test-slave-vdtf krb5kdc[18878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:46.253600 18985 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:49:46.254581 18985 webserver.cc:460] Webserver started at http://127.0.201.129:36845/ using document root <none> and password file <none>
I20260925 22:49:46.255128 18985 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:46.255218 18985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:46.255440 18985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:46.257453 18985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "4c8507ab49a74d929ce1b0e96055588f"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-vdtf"
server_key: "e007d8d351f6d8276c948aa0af36534a"
server_key_iv: "d4e57d7bd203a94948a2452ecf42e8fa"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.258005 18985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "4c8507ab49a74d929ce1b0e96055588f"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-vdtf"
server_key: "e007d8d351f6d8276c948aa0af36534a"
server_key_iv: "d4e57d7bd203a94948a2452ecf42e8fa"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.261675 18985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:46.264027 19000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.265192 18985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:46.265328 18985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "4c8507ab49a74d929ce1b0e96055588f"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-vdtf"
server_key: "e007d8d351f6d8276c948aa0af36534a"
server_key_iv: "d4e57d7bd203a94948a2452ecf42e8fa"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.265434 18985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:46.276777 18985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:46.280025 18985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:46.280215 18985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:46.280751 18985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:46.281517 18985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:46.281589 18985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.281653 18985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:46.281701 18985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.290278 18985 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:35159
I20260925 22:49:46.290305 19113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:35159 every 8 connection(s)
I20260925 22:49:46.291141 18985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:46.291652 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 18985
I20260925 22:49:46.291805 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:46.292131 806 external_mini_cluster.cc:1428] Setting key ca2df2f97bdcf20d46bea08a851c7960
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
Sep 25 22:49:46 dist-test-slave-vdtf krb5kdc[18878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:49:46.306682 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.292946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:39815 (local address 127.0.201.190:40737)
0925 22:49:46.293091 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.293096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.294068 (+ 972us) server_negotiation.cc:407] Connection header received
0925 22:49:46.295639 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.295642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.295732 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.295810 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:46.297425 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.298023 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.299928 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.300166 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.300247 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.303035 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:46.303038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:46.303040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:46.303063 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:46.304425 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.304978 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.304983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.304986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.305077 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.305445 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.305447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.305449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.305645 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:46.305720 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.306365 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.306468 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:49:46.307590 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.293428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40737 (local address 127.0.201.129:39815)
0925 22:49:46.293890 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.293944 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.295396 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.296021 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.296031 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.296555 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:46.297258 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.297281 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.298159 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.298166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.299805 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.299821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.300329 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.300336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.300563 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.301163 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:46.301187 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:46.302876 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:46.304533 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.304538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.304549 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.304845 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.305162 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.305164 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.305165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.305341 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.305754 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:46.305764 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:46.306116 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.306335 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.306722 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":127,"threads_started":1}
I20260925 22:49:46.308929 19114 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40737
I20260925 22:49:46.309172 19114 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:46.309654 19114 heartbeater.cc:511] Master 127.0.201.190:40737 requested a full tablet report, sending...
I20260925 22:49:46.311159 18923 ts_manager.cc:194] Registered new tserver with Master: 4c8507ab49a74d929ce1b0e96055588f (127.0.201.129:35159)
I20260925 22:49:46.312304 18923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:39815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:49:46.320921 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40737
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:46.431638 19119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:46.431978 19119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:46.432056 19119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:46.436177 19119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:46.436256 19119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:46.436347 19119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:49:46.441877 19119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:46.442997 19119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:46.444072 19119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:46.453821 19124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.453926 19119 server_base.cc:1034] running on GCE node
W20260925 22:49:46.453831 19125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:46.453860 19127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.454448 19119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:46.455137 19119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:46.456326 19119 hybrid_clock.cc:648] HybridClock initialized: now 1790376586456310 us; error 27 us; skew 500 ppm
Sep 25 22:49:46 dist-test-slave-vdtf krb5kdc[18878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:46.459071 19119 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:49:46.460057 19119 webserver.cc:460] Webserver started at http://127.0.201.130:37607/ using document root <none> and password file <none>
I20260925 22:49:46.460620 19119 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:46.460713 19119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:46.460947 19119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:46.462977 19119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "40c619fe78f14c42882023f8251b6459"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-vdtf"
server_key: "91ad5d2d616ea81e6e2829479e08f517"
server_key_iv: "314854dc27a25d1cb9a1de3cc7bec6e3"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.463554 19119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "40c619fe78f14c42882023f8251b6459"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-vdtf"
server_key: "91ad5d2d616ea81e6e2829479e08f517"
server_key_iv: "314854dc27a25d1cb9a1de3cc7bec6e3"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.467439 19119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:49:46.469945 19134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.471238 19119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:46.471397 19119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "40c619fe78f14c42882023f8251b6459"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-vdtf"
server_key: "91ad5d2d616ea81e6e2829479e08f517"
server_key_iv: "314854dc27a25d1cb9a1de3cc7bec6e3"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.471514 19119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:46.483335 19119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:46.486487 19119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:46.486680 19119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:46.487237 19119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:46.488139 19119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:46.488206 19119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.488269 19119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:46.488315 19119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.496949 19119 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:34021
I20260925 22:49:46.496974 19247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:34021 every 8 connection(s)
I20260925 22:49:46.497771 19119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
Sep 25 22:49:46 dist-test-slave-vdtf krb5kdc[18878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:46.507191 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 19119
I20260925 22:49:46.507320 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:46.507627 806 external_mini_cluster.cc:1428] Setting key bb8777074b4482344402036db422df3d
I20260925 22:49:46.510824 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.499380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:60165 (local address 127.0.201.190:40737)
0925 22:49:46.499511 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.499516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.500331 (+ 815us) server_negotiation.cc:407] Connection header received
0925 22:49:46.501559 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.501563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.501621 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.501697 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:46.502772 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.503311 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.504489 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.504727 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.504840 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.507157 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:46.507165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:46.507167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:46.507191 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:46.508724 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.509273 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.509277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.509278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.509386 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.509698 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.509705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.509710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.509960 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:46.510062 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.510487 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.510624 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:49:46.511762 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.499790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40737 (local address 127.0.201.130:60165)
0925 22:49:46.500195 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.500235 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.501353 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.501854 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.501860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.502190 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:46.502633 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.502645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.503431 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.503434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.504360 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.504369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.504873 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.504877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.505046 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.505589 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:46.505605 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:46.506995 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:46.508909 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.508913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.508924 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.509165 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.509478 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.509479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.509481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.509603 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.510060 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:46.510064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:46.510263 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.510407 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.510704 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20260925 22:49:46.512912 19248 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40737
I20260925 22:49:46.513151 19248 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:46.513629 19248 heartbeater.cc:511] Master 127.0.201.190:40737 requested a full tablet report, sending...
I20260925 22:49:46.514631 18923 ts_manager.cc:194] Registered new tserver with Master: 40c619fe78f14c42882023f8251b6459 (127.0.201.130:34021)
I20260925 22:49:46.515218 18923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:60165
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:49:46.534793 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40737
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:46.645476 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:46.645732 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:46.645773 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:46.649785 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:46.649835 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:46.649930 19253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:49:46.655248 19253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:46.656275 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:46.657191 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:46.666597 19261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:46.666628 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.666771 19253 server_base.cc:1034] running on GCE node
W20260925 22:49:46.666628 19259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.667443 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:46.668131 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:46.669346 19253 hybrid_clock.cc:648] HybridClock initialized: now 1790376586669294 us; error 52 us; skew 500 ppm
Sep 25 22:49:46 dist-test-slave-vdtf krb5kdc[18878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:46.672137 19253 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:49:46.673012 19253 webserver.cc:460] Webserver started at http://127.0.201.131:46599/ using document root <none> and password file <none>
I20260925 22:49:46.673560 19253 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:46.673645 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:46.673871 19253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:46.676074 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "76986f2db92e4cd39eb143b797d9263c"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-vdtf"
server_key: "9005a8fb61701814040f480b752504ef"
server_key_iv: "b1377e11a830a0d0e56f9b294cc993a2"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.676644 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "76986f2db92e4cd39eb143b797d9263c"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-vdtf"
server_key: "9005a8fb61701814040f480b752504ef"
server_key_iv: "b1377e11a830a0d0e56f9b294cc993a2"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.680186 19253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:46.682742 19268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.684000 19253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:46.684118 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "76986f2db92e4cd39eb143b797d9263c"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-vdtf"
server_key: "9005a8fb61701814040f480b752504ef"
server_key_iv: "b1377e11a830a0d0e56f9b294cc993a2"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.684221 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:46.699801 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:46.702521 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:46.702724 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:46.703298 19253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:46.704164 19253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:46.704231 19253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.704300 19253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:46.704349 19253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.712937 19253 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:38875
I20260925 22:49:46.713001 19381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:38875 every 8 connection(s)
I20260925 22:49:46.713817 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:46.720759 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 19253
I20260925 22:49:46.720896 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:46.721220 806 external_mini_cluster.cc:1428] Setting key ba2f82d14b5a323e2e2562215f0f2ec5
Sep 25 22:49:46 dist-test-slave-vdtf krb5kdc[18878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:46.727464 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.715563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:53027 (local address 127.0.201.190:40737)
0925 22:49:46.715772 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.715776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.716538 (+ 762us) server_negotiation.cc:407] Connection header received
0925 22:49:46.717832 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.717836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.717894 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.717975 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:46.719101 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.719630 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.720998 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.721313 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.721434 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.724121 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:46.724124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:46.724126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:46.724151 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:46.725668 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.726099 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.726101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.726102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.726175 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.726498 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.726501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.726502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.726689 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:46.726776 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.727234 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.727347 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:49:46.728250 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.715978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40737 (local address 127.0.201.131:53027)
0925 22:49:46.716386 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.716426 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.717620 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.718135 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.718141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.718484 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:46.718955 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.718968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.719773 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.719779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.720795 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.720807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.721489 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.721493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.721669 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.722332 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:46.722350 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:46.723968 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:46.725763 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.725767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.725777 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.726012 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.726253 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.726255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.726256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.726374 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.726781 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:46.726786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:46.727027 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.727176 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.727409 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":94,"threads_started":1}
I20260925 22:49:46.729451 19382 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40737
I20260925 22:49:46.729696 19382 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:46.730159 19382 heartbeater.cc:511] Master 127.0.201.190:40737 requested a full tablet report, sending...
I20260925 22:49:46.731233 18923 ts_manager.cc:194] Registered new tserver with Master: 76986f2db92e4cd39eb143b797d9263c (127.0.201.131:38875)
I20260925 22:49:46.731871 18923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:53027
I20260925 22:49:46.735347 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:49:46.744522 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.737269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.0.201.190:40737)
0925 22:49:46.737399 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.737404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.737577 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:49:46.737666 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.737668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.737704 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.737777 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:46.738416 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.738929 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.739789 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.739990 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.740066 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.741705 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:46.741708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:46.741710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:46.741729 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:46.742938 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.743370 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.743372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.743373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.743439 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.743740 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.743742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.743744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.743918 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:46.744027 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.744241 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.744338 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:49:46.747242 18923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:46.750186 18923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:46.764940 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.758848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:34021 (local address 127.0.0.1:55672)
0925 22:49:46.759773 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.759788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.759874 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.760429 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.760434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.760453 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:46.760596 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.760607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.762083 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.762086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.763340 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.763349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.764420 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.764424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.764522 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.764577 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.764680 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.764795 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":104,"thread_start_us":102,"threads_started":1}
I20260925 22:49:46.764977 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.758989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.0.201.130:34021)
0925 22:49:46.759727 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.759739 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.759989 (+ 250us) server_negotiation.cc:407] Connection header received
0925 22:49:46.760090 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.760096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.760305 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.760436 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:46.760734 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.761920 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.763458 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.764291 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.764399 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.764537 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.764724 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.764823 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":113,"threads_started":1}
I20260925 22:49:46.765144 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.758735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:35159 (local address 127.0.0.1:58174)
0925 22:49:46.759469 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.759529 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.759751 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.760678 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.760681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.760699 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:46.760804 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.760822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.762081 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.762087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.763795 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.763806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.764804 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.764807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.764892 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.764912 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.764978 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.765041 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":287,"spinlock_wait_cycles":16000,"thread_start_us":70,"threads_started":1}
I20260925 22:49:46.765707 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.758845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.0.201.129:35159)
0925 22:49:46.759249 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.759260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.759620 (+ 360us) server_negotiation.cc:407] Connection header received
0925 22:49:46.760339 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.760345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.760529 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.760695 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:46.760904 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.761946 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.763930 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.764514 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.764610 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.764748 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.765417 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.765571 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":95,"threads_started":1}
I20260925 22:49:46.768006 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.759899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.0.201.131:38875)
0925 22:49:46.760703 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.760714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.760734 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:49:46.760804 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.760820 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.760988 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.761127 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:46.761299 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.761978 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.763913 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.764712 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.764808 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.764925 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.767777 (+ 2852us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.767879 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":151,"threads_started":1}
I20260925 22:49:46.767719 19182 tablet_service.cc:1467] Processing CreateTablet for tablet 92115c990e854948b881e60a4e97ec2e (DEFAULT_TABLE table=test-table [id=ab078335c83047c4950762baa26b24b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:46.767865 19048 tablet_service.cc:1467] Processing CreateTablet for tablet 92115c990e854948b881e60a4e97ec2e (DEFAULT_TABLE table=test-table [id=ab078335c83047c4950762baa26b24b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:46.768308 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.759592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:38875 (local address 127.0.0.1:47270)
0925 22:49:46.760172 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.760184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.760282 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.761118 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.761121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.761137 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:46.761211 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.761216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.762140 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.762142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.763795 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.763806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.764990 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.764996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.765103 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.765133 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.765211 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.768189 (+ 2978us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":182,"thread_start_us":254,"threads_started":1}
I20260925 22:49:46.768748 19182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92115c990e854948b881e60a4e97ec2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:46.768867 19048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92115c990e854948b881e60a4e97ec2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:46.770939 19316 tablet_service.cc:1467] Processing CreateTablet for tablet 92115c990e854948b881e60a4e97ec2e (DEFAULT_TABLE table=test-table [id=ab078335c83047c4950762baa26b24b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:46.771910 19316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92115c990e854948b881e60a4e97ec2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:46.775895 19400 tablet_bootstrap.cc:492] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459: Bootstrap starting.
I20260925 22:49:46.775933 19399 tablet_bootstrap.cc:492] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f: Bootstrap starting.
I20260925 22:49:46.778046 19399 tablet_bootstrap.cc:654] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:46.778324 19401 tablet_bootstrap.cc:492] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: Bootstrap starting.
I20260925 22:49:46.778445 19400 tablet_bootstrap.cc:654] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:46.779458 19400 log.cc:824] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:46.779458 19399 log.cc:824] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:46.780539 19401 tablet_bootstrap.cc:654] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:46.781188 19401 log.cc:824] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:46.782858 19401 tablet_bootstrap.cc:492] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: No bootstrap required, opened a new log
I20260925 22:49:46.782963 19400 tablet_bootstrap.cc:492] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459: No bootstrap required, opened a new log
I20260925 22:49:46.783058 19401 ts_tablet_manager.cc:1397] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:49:46.783144 19400 ts_tablet_manager.cc:1397] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:49:46.783166 19399 tablet_bootstrap.cc:492] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f: No bootstrap required, opened a new log
I20260925 22:49:46.783299 19399 ts_tablet_manager.cc:1397] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:49:46.786307 19401 raft_consensus.cc:348] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8507ab49a74d929ce1b0e96055588f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35159 } } peers { permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 } } peers { permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 } }
I20260925 22:49:46.786540 19401 raft_consensus.cc:374] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:46.786589 19401 raft_consensus.cc:729] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76986f2db92e4cd39eb143b797d9263c, State: Initialized, Role: FOLLOWER
I20260925 22:49:46.787021 19401 consensus_queue.cc:260] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8507ab49a74d929ce1b0e96055588f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35159 } } peers { permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 } } peers { permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 } }
I20260925 22:49:46.786948 19399 raft_consensus.cc:348] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8507ab49a74d929ce1b0e96055588f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35159 } } peers { permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 } } peers { permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 } }
I20260925 22:49:46.786948 19400 raft_consensus.cc:348] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8507ab49a74d929ce1b0e96055588f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35159 } } peers { permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 } } peers { permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 } }
I20260925 22:49:46.787168 19400 raft_consensus.cc:374] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:46.787168 19399 raft_consensus.cc:374] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:46.787223 19400 raft_consensus.cc:729] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c619fe78f14c42882023f8251b6459, State: Initialized, Role: FOLLOWER
I20260925 22:49:46.787230 19399 raft_consensus.cc:729] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8507ab49a74d929ce1b0e96055588f, State: Initialized, Role: FOLLOWER
I20260925 22:49:46.787647 19382 heartbeater.cc:503] Master 127.0.201.190:40737 was elected leader, sending a full tablet report...
I20260925 22:49:46.787642 19400 consensus_queue.cc:260] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8507ab49a74d929ce1b0e96055588f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35159 } } peers { permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 } } peers { permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 } }
I20260925 22:49:46.787698 19399 consensus_queue.cc:260] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8507ab49a74d929ce1b0e96055588f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35159 } } peers { permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 } } peers { permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 } }
I20260925 22:49:46.788043 19401 ts_tablet_manager.cc:1428] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:49:46.788534 19399 ts_tablet_manager.cc:1428] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:49:46.788540 19400 ts_tablet_manager.cc:1428] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:49:46.788877 19248 heartbeater.cc:503] Master 127.0.201.190:40737 was elected leader, sending a full tablet report...
I20260925 22:49:46.788887 19114 heartbeater.cc:503] Master 127.0.201.190:40737 was elected leader, sending a full tablet report...
W20260925 22:49:46.792363 19115 tablet.cc:2369] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:46.957216 19406 raft_consensus.cc:482] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:46.957470 19406 raft_consensus.cc:504] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8507ab49a74d929ce1b0e96055588f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35159 } } peers { permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 } } peers { permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 } }
I20260925 22:49:46.958689 19406 leader_election.cc:288] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c619fe78f14c42882023f8251b6459 (127.0.201.130:34021), 76986f2db92e4cd39eb143b797d9263c (127.0.201.131:38875)
I20260925 22:49:46.964079 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.959106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:49651 (local address 127.0.201.131:38875)
0925 22:49:46.959236 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.959240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.959418 (+ 178us) server_negotiation.cc:407] Connection header received
0925 22:49:46.959517 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.959519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.959563 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.959651 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:46.960147 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.960923 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.962785 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.963403 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.963527 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.963577 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.963885 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.963952 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:49:46.964079 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.959017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:53595 (local address 127.0.201.130:34021)
0925 22:49:46.959159 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.959163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.959338 (+ 175us) server_negotiation.cc:407] Connection header received
0925 22:49:46.959499 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.959512 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.959568 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.959653 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:46.960147 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.960923 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.962785 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.963403 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.963530 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.963577 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.963885 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.963952 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:49:46.964211 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.958981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:38875 (local address 127.0.201.129:49651)
0925 22:49:46.959327 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.959351 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.959437 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.959704 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.959707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.959750 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:46.960027 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.960035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.961044 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.961048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.962653 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.962665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.963533 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.963537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.963705 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.963763 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.963862 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.963996 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":348,"spinlock_wait_cycles":7168,"thread_start_us":138,"threads_started":1}
I20260925 22:49:46.964210 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.958900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:34021 (local address 127.0.201.129:53595)
0925 22:49:46.959235 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.959250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.959371 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.959704 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.959707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.959748 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:46.960030 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.960035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.961044 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.961048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.962653 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.962665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.963533 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.963537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.963701 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.963766 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.963867 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.963996 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":236,"spinlock_wait_cycles":36224,"thread_start_us":82,"threads_started":1}
W20260925 22:49:46.964742 19383 tablet.cc:2369] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:46.964958 19336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92115c990e854948b881e60a4e97ec2e" candidate_uuid: "4c8507ab49a74d929ce1b0e96055588f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76986f2db92e4cd39eb143b797d9263c" is_pre_election: true
I20260925 22:49:46.964956 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92115c990e854948b881e60a4e97ec2e" candidate_uuid: "4c8507ab49a74d929ce1b0e96055588f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c619fe78f14c42882023f8251b6459" is_pre_election: true
I20260925 22:49:46.965261 19336 raft_consensus.cc:2454] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8507ab49a74d929ce1b0e96055588f in term 0.
I20260925 22:49:46.965261 19202 raft_consensus.cc:2454] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8507ab49a74d929ce1b0e96055588f in term 0.
I20260925 22:49:46.965716 19001 leader_election.cc:302] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c619fe78f14c42882023f8251b6459, 4c8507ab49a74d929ce1b0e96055588f; no voters:
I20260925 22:49:46.966010 19406 raft_consensus.cc:2781] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:46.966104 19406 raft_consensus.cc:482] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:46.966161 19406 raft_consensus.cc:3037] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:46.967183 19406 raft_consensus.cc:504] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8507ab49a74d929ce1b0e96055588f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35159 } } peers { permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 } } peers { permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 } }
I20260925 22:49:46.967554 19406 leader_election.cc:288] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [CANDIDATE]: Term 1 election: Requested vote from peers 40c619fe78f14c42882023f8251b6459 (127.0.201.130:34021), 76986f2db92e4cd39eb143b797d9263c (127.0.201.131:38875)
I20260925 22:49:46.967963 19336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92115c990e854948b881e60a4e97ec2e" candidate_uuid: "4c8507ab49a74d929ce1b0e96055588f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76986f2db92e4cd39eb143b797d9263c"
I20260925 22:49:46.967974 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92115c990e854948b881e60a4e97ec2e" candidate_uuid: "4c8507ab49a74d929ce1b0e96055588f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c619fe78f14c42882023f8251b6459"
I20260925 22:49:46.968101 19336 raft_consensus.cc:3037] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:46.968098 19202 raft_consensus.cc:3037] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:46.969192 19336 raft_consensus.cc:2454] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8507ab49a74d929ce1b0e96055588f in term 1.
I20260925 22:49:46.969305 19202 raft_consensus.cc:2454] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8507ab49a74d929ce1b0e96055588f in term 1.
I20260925 22:49:46.969563 19002 leader_election.cc:302] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8507ab49a74d929ce1b0e96055588f, 76986f2db92e4cd39eb143b797d9263c; no voters:
I20260925 22:49:46.969756 19406 raft_consensus.cc:2781] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:46.970000 19406 raft_consensus.cc:686] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8507ab49a74d929ce1b0e96055588f, State: Running, Role: LEADER
I20260925 22:49:46.970351 19406 consensus_queue.cc:237] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8507ab49a74d929ce1b0e96055588f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35159 } } peers { permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 } } peers { permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 } }
I20260925 22:49:46.973441 18920 catalog_manager.cc:5495] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8507ab49a74d929ce1b0e96055588f (127.0.201.129). New cstate: current_term: 1 leader_uuid: "4c8507ab49a74d929ce1b0e96055588f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8507ab49a74d929ce1b0e96055588f" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 35159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 } health_report { overall_health: UNKNOWN } } }
W20260925 22:49:46.998816 19249 tablet.cc:2369] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:47.006342 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.002747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.0.201.129:35159)
0925 22:49:47.002897 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.002901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.002916 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:49:47.003056 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.003058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.003112 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.003230 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:47.003486 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.004010 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.004877 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.005093 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.005207 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.005511 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:47.005901 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:47.006064 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.006168 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.006207 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:49:47.012120 19202 raft_consensus.cc:1261] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459 [term 1 FOLLOWER]: Refusing update from remote peer 4c8507ab49a74d929ce1b0e96055588f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:47.012141 19336 raft_consensus.cc:1261] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c [term 1 FOLLOWER]: Refusing update from remote peer 4c8507ab49a74d929ce1b0e96055588f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:47.012791 19410 consensus_queue.cc:1035] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [LEADER]: Connected to new peer: Peer: permanent_uuid: "76986f2db92e4cd39eb143b797d9263c" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 38875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:47.012976 19406 consensus_queue.cc:1035] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c619fe78f14c42882023f8251b6459" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 34021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:47.022356 19414 mvcc.cc:205] Tried to move back new op lower bound from 7333382500393738240 to 7333382500233048064. Current Snapshot: MvccSnapshot[applied={T|T < 7333382500393738240 or (T in {7333382500393738240})}]
I20260925 22:49:47.022481 19415 mvcc.cc:205] Tried to move back new op lower bound from 7333382500393738240 to 7333382500233048064. Current Snapshot: MvccSnapshot[applied={T|T < 7333382500393738240}]
I20260925 22:49:47.031203 18920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:47.031378 18920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:47.033567 18920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69b0f7fc81f44af5ad4bd4640704bf66: Sending DeleteTablet for 3 replicas of tablet 92115c990e854948b881e60a4e97ec2e
I20260925 22:49:47.034271 19182 tablet_service.cc:1514] Processing DeleteTablet for tablet 92115c990e854948b881e60a4e97ec2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:47 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:55672
I20260925 22:49:47.034308 19048 tablet_service.cc:1514] Processing DeleteTablet for tablet 92115c990e854948b881e60a4e97ec2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:47 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:58174
I20260925 22:49:47.034370 19316 tablet_service.cc:1514] Processing DeleteTablet for tablet 92115c990e854948b881e60a4e97ec2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:47 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:47270
I20260925 22:49:47.034837 19423 tablet_replica.cc:330] T 92115c990e854948b881e60a4e97ec2e P 4c8507ab49a74d929ce1b0e96055588f: stopping tablet replica
I20260925 22:49:47.035035 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 18985
I20260925 22:49:47.039458 19424 tablet_replica.cc:330] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459: stopping tablet replica
I20260925 22:49:47.039757 19425 tablet_replica.cc:330] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: stopping tablet replica
I20260925 22:49:47.039847 19424 raft_consensus.cc:2229] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:47.040050 19425 raft_consensus.cc:2229] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:47.040192 19424 raft_consensus.cc:2258] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:47.040289 19425 raft_consensus.cc:2258] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:47.041712 19425 ts_tablet_manager.cc:1905] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:47.041712 19424 ts_tablet_manager.cc:1905] T 92115c990e854948b881e60a4e97ec2e P 40c619fe78f14c42882023f8251b6459: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:49:47.042312 18908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:35159 (error 108)
I20260925 22:49:47.042570 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 19119
I20260925 22:49:47.043917 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.043270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:35159 (local address 127.0.0.1:58186)
0925 22:49:47.043749 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:47.043841 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372}
W20260925 22:49:47.044420 18908 catalog_manager.cc:4584] TS 4c8507ab49a74d929ce1b0e96055588f (127.0.201.129:35159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92115c990e854948b881e60a4e97ec2e: Network error: Client connection negotiation failed: client connection to 127.0.201.129:35159: connect: Connection refused (error 111)
I20260925 22:49:47.045960 19425 ts_tablet_manager.cc:1918] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:47.046069 19425 log.cc:1196] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/92115c990e854948b881e60a4e97ec2e
I20260925 22:49:47.046370 19425 ts_tablet_manager.cc:1939] T 92115c990e854948b881e60a4e97ec2e P 76986f2db92e4cd39eb143b797d9263c: Deleting consensus metadata
I20260925 22:49:47.047410 18908 catalog_manager.cc:4841] TS 76986f2db92e4cd39eb143b797d9263c (127.0.201.131:38875): tablet 92115c990e854948b881e60a4e97ec2e (table test-table [id=ab078335c83047c4950762baa26b24b1]) successfully deleted
I20260925 22:49:47.048610 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 19253
I20260925 22:49:47.049948 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.048749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:34021 (local address 127.0.0.1:55678)
0925 22:49:47.049796 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:47.049861 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.130:34021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":952}
I20260925 22:49:47.054414 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 18891
2026-09-25T22:49:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2404 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T22:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:47Z Disabled control of system clock
I20260925 22:49:47.084542 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40179
--webserver_interface=127.0.201.190
--webserver_port=0
--builtin_ntp_servers=127.0.201.148:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:49:47.192749 19430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:47.192989 19430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:47.193023 19430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:47.197027 19430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:47.197078 19430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:47.197094 19430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:47.197148 19430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:47.197176 19430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:47.201901 19430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:40179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:47.202881 19430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:47.204212 19430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:47.212222 19436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:47.212260 19438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:47.212224 19435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:47.212569 19430 server_base.cc:1034] running on GCE node
I20260925 22:49:47.213120 19430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:47.214151 19430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:47.215337 19430 hybrid_clock.cc:648] HybridClock initialized: now 1790376587215311 us; error 45 us; skew 500 ppm
I20260925 22:49:47.217171 19430 webserver.cc:460] Webserver started at http://127.0.201.190:39079/ using document root <none> and password file <none>
I20260925 22:49:47.217729 19430 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:47.217819 19430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:47.218036 19430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:47.220060 19430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "34b39c5535e849d89fd9b5ac2bb7667a"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "94c222168f76b3a24cb8d47867aa1ef9"
server_key_iv: "5f45875c9eb1e5aab2bda7638c573040"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.220602 19430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "34b39c5535e849d89fd9b5ac2bb7667a"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "94c222168f76b3a24cb8d47867aa1ef9"
server_key_iv: "5f45875c9eb1e5aab2bda7638c573040"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.224080 19430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:47.226300 19444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:47.227447 19430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:47.227589 19430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "34b39c5535e849d89fd9b5ac2bb7667a"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "94c222168f76b3a24cb8d47867aa1ef9"
server_key_iv: "5f45875c9eb1e5aab2bda7638c573040"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.227712 19430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:47.240811 19430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:47.241458 19430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:47.241647 19430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:47.248382 19430 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:40179
I20260925 22:49:47.248399 19496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:40179 every 8 connection(s)
I20260925 22:49:47.249346 19430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:47.249713 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 19430
I20260925 22:49:47.249840 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:47.250191 806 external_mini_cluster.cc:1428] Setting key bee8083ca55c99886692fe524d8034d3
I20260925 22:49:47.252703 19497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:47.259239 19497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a: Bootstrap starting.
I20260925 22:49:47.260432 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.251578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.0.201.190:40179)
0925 22:49:47.252234 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.252254 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.252294 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:49:47.253145 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.253169 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.253644 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.253991 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:47.254419 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.255907 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.256892 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.257587 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.257744 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.258508 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:47.258514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:47.258529 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:47.258990 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:47.259057 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:47.259082 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:47.259180 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.259271 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.259530 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":148,"threads_started":1}
I20260925 22:49:47.261763 19497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:47.262589 19497 log.cc:824] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:47.264423 19497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a: No bootstrap required, opened a new log
I20260925 22:49:47.267917 19497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b39c5535e849d89fd9b5ac2bb7667a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40179 } }
I20260925 22:49:47.268152 19497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:47.268230 19497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b39c5535e849d89fd9b5ac2bb7667a, State: Initialized, Role: FOLLOWER
I20260925 22:49:47.268636 19497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b39c5535e849d89fd9b5ac2bb7667a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40179 } }
I20260925 22:49:47.268821 19497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:47.268908 19497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:47.269052 19497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:47.269897 19497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b39c5535e849d89fd9b5ac2bb7667a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40179 } }
I20260925 22:49:47.270218 19497 leader_election.cc:302] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b39c5535e849d89fd9b5ac2bb7667a; no voters:
I20260925 22:49:47.270516 19497 leader_election.cc:288] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:47.270643 19502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:47.270866 19502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [term 1 LEADER]: Becoming Leader. State: Replica: 34b39c5535e849d89fd9b5ac2bb7667a, State: Running, Role: LEADER
I20260925 22:49:47.271183 19502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b39c5535e849d89fd9b5ac2bb7667a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40179 } }
I20260925 22:49:47.271714 19497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:47.272939 19504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b39c5535e849d89fd9b5ac2bb7667a. Latest consensus state: current_term: 1 leader_uuid: "34b39c5535e849d89fd9b5ac2bb7667a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b39c5535e849d89fd9b5ac2bb7667a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40179 } } }
I20260925 22:49:47.272950 19503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b39c5535e849d89fd9b5ac2bb7667a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b39c5535e849d89fd9b5ac2bb7667a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 40179 } } }
I20260925 22:49:47.273084 19504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:47.273084 19503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:47.273447 19511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:47.276298 19511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:47.281598 19511 catalog_manager.cc:1347] Generated new cluster ID: d5cc70d845774665aa6b82798decd7c3
I20260925 22:49:47.281663 19511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:47.297381 19511 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:47.298627 19511 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:47.303157 19511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a: Generated new TSK 0
I20260925 22:49:47.303987 19511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:47.312729 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40179
--builtin_ntp_servers=127.0.201.148:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:49:47.421527 19521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:47.421801 19521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:47.421875 19521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:47.425822 19521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:47.425899 19521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:47.425990 19521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:49:47.430658 19521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:47.431766 19521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:47.433107 19521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:47.442555 19526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:47.442579 19527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:47.442636 19521 server_base.cc:1034] running on GCE node
W20260925 22:49:47.442587 19529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:47.443143 19521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:47.443817 19521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:47.444978 19521 hybrid_clock.cc:648] HybridClock initialized: now 1790376587444966 us; error 23 us; skew 500 ppm
I20260925 22:49:47.446653 19521 webserver.cc:460] Webserver started at http://127.0.201.129:35635/ using document root <none> and password file <none>
I20260925 22:49:47.447155 19521 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:47.447211 19521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:47.447360 19521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:47.449353 19521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "ba17e9134298492e954b3bc574f1d38b"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "21634bed5d437d89cdcbf7d80b7ef9bf"
server_key_iv: "fcd3cbcd178a2e3c47aea491c08eb933"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.449836 19521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "ba17e9134298492e954b3bc574f1d38b"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "21634bed5d437d89cdcbf7d80b7ef9bf"
server_key_iv: "fcd3cbcd178a2e3c47aea491c08eb933"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.453245 19521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:49:47.455484 19535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:47.456693 19521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:47.456823 19521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "ba17e9134298492e954b3bc574f1d38b"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "21634bed5d437d89cdcbf7d80b7ef9bf"
server_key_iv: "fcd3cbcd178a2e3c47aea491c08eb933"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.456955 19521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:47.474722 19521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:47.475340 19521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:47.475528 19521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:47.476177 19521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:47.477095 19521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:47.477160 19521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:47.477224 19521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:47.477272 19521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:47.486100 19521 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:44827
I20260925 22:49:47.486123 19648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:44827 every 8 connection(s)
I20260925 22:49:47.486928 19521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:47.488945 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 19521
I20260925 22:49:47.489071 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:47.489357 806 external_mini_cluster.cc:1428] Setting key 0b4961c7776957a3e7e1ddf22154d395
I20260925 22:49:47.491447 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40179
--builtin_ntp_servers=127.0.201.148:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:49:47.497627 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.488686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:40487 (local address 127.0.201.190:40179)
0925 22:49:47.488888 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.488893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.490089 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 22:49:47.491820 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.491823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.491869 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.491952 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:47.493132 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.493679 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.495466 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.495765 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.495869 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.497160 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:47.497163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:47.497165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:47.497176 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:47.497194 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:47.497202 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:47.497258 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.497450 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.497509 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 22:49:47.498867 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.489173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40179 (local address 127.0.201.129:40487)
0925 22:49:47.489885 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:47.489953 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:47.491586 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:47.492149 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:47.492160 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:47.492272 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:47.492979 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.492996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.493801 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.493805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.495324 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.495339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.495942 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.495950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.496195 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.496852 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:47.496856 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:47.496914 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:47.497004 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:47.497290 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:47.497302 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:47.497310 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:49:47.497449 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 22:49:47.497758 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260925 22:49:47.500046 19649 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40179
I20260925 22:49:47.500296 19649 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:47.500936 19649 heartbeater.cc:511] Master 127.0.201.190:40179 requested a full tablet report, sending...
I20260925 22:49:47.502406 19461 ts_manager.cc:194] Registered new tserver with Master: ba17e9134298492e954b3bc574f1d38b (127.0.201.129:44827)
I20260925 22:49:47.503510 19461 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.129:40487
W20260925 22:49:47.604504 19652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:47.604785 19652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:47.604838 19652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:47.609366 19652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:47.609442 19652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:47.609525 19652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:49:47.614183 19652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:47.615213 19652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:47.616629 19652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:47.625985 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:47.626024 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:47.626034 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:47.626120 19652 server_base.cc:1034] running on GCE node
I20260925 22:49:47.626588 19652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:47.627194 19652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:47.628373 19652 hybrid_clock.cc:648] HybridClock initialized: now 1790376587628362 us; error 24 us; skew 500 ppm
I20260925 22:49:47.630468 19652 webserver.cc:460] Webserver started at http://127.0.201.130:40467/ using document root <none> and password file <none>
I20260925 22:49:47.631108 19652 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:47.631196 19652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:47.631413 19652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:47.633671 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "d9a71311a5b8451ba764732010644819"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "6e38f84ea90f73db345eb30113275f0c"
server_key_iv: "e6b1add6703a22e93a653dff6616f925"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.634210 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "d9a71311a5b8451ba764732010644819"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "6e38f84ea90f73db345eb30113275f0c"
server_key_iv: "e6b1add6703a22e93a653dff6616f925"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.637683 19652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:47.639973 19666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:47.641109 19652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:47.641237 19652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "d9a71311a5b8451ba764732010644819"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "6e38f84ea90f73db345eb30113275f0c"
server_key_iv: "e6b1add6703a22e93a653dff6616f925"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.641341 19652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:47.673873 19652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:47.674566 19652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:47.674764 19652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:47.675350 19652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:47.676357 19652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:47.676435 19652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:47.676503 19652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:47.676548 19652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:47.685351 19652 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:45213
I20260925 22:49:47.685388 19779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:45213 every 8 connection(s)
I20260925 22:49:47.686228 19652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:47.688572 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 19652
I20260925 22:49:47.688691 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:47.689014 806 external_mini_cluster.cc:1428] Setting key 4412d264832559f11e74992b390d7526
I20260925 22:49:47.690855 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40179
--builtin_ntp_servers=127.0.201.148:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:49:47.695927 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.687885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:49625 (local address 127.0.201.190:40179)
0925 22:49:47.688031 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.688034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.689246 (+ 1212us) server_negotiation.cc:407] Connection header received
0925 22:49:47.690613 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.690616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.690666 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.690772 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:47.691702 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.692499 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.693657 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.694078 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.694210 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.695320 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:47.695325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:47.695330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:47.695347 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:47.695375 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:47.695388 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:47.695478 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.695617 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.695724 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:49:47.696645 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.688321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40179 (local address 127.0.201.130:49625)
0925 22:49:47.689082 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:47.689137 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:47.690402 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:47.690949 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:47.690961 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:47.691080 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:47.691545 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.691557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.692644 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.692647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.693532 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.693542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.694275 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.694282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.694446 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.695066 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:47.695068 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:47.695106 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:47.695168 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:47.695481 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:47.695487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:47.695491 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:49:47.695583 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:49:47.695845 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":144,"threads_started":1}
I20260925 22:49:47.697834 19780 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40179
I20260925 22:49:47.698076 19780 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:47.698557 19780 heartbeater.cc:511] Master 127.0.201.190:40179 requested a full tablet report, sending...
I20260925 22:49:47.699575 19461 ts_manager.cc:194] Registered new tserver with Master: d9a71311a5b8451ba764732010644819 (127.0.201.130:45213)
I20260925 22:49:47.700165 19461 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.130:49625
W20260925 22:49:47.802614 19783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:47.802903 19783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:47.802973 19783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:47.807029 19783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:47.807116 19783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:47.807209 19783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:49:47.812177 19783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:47.813280 19783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:47.814666 19783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:47.824234 19791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:47.824254 19789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:47.824254 19788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:47.824487 19783 server_base.cc:1034] running on GCE node
I20260925 22:49:47.824888 19783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:47.825568 19783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:47.826761 19783 hybrid_clock.cc:648] HybridClock initialized: now 1790376587826744 us; error 30 us; skew 500 ppm
I20260925 22:49:47.828629 19783 webserver.cc:460] Webserver started at http://127.0.201.131:45125/ using document root <none> and password file <none>
I20260925 22:49:47.829226 19783 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:47.829319 19783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:47.829566 19783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:47.831801 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "28634657630b4493ad878e4ac5008e76"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "4c26a069b0871d86fe2a0a66759a4652"
server_key_iv: "50381e15a004edae592e66a86f83609c"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.832401 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "28634657630b4493ad878e4ac5008e76"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "4c26a069b0871d86fe2a0a66759a4652"
server_key_iv: "50381e15a004edae592e66a86f83609c"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.836115 19783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:47.838670 19797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:47.840432 19783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:47.840560 19783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "28634657630b4493ad878e4ac5008e76"
format_stamp: "Formatted at 2026-09-25 22:49:47 on dist-test-slave-vdtf"
server_key: "4c26a069b0871d86fe2a0a66759a4652"
server_key_iv: "50381e15a004edae592e66a86f83609c"
server_key_version: "encryptionkey@0"
I20260925 22:49:47.840698 19783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:47.852807 19783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:47.853446 19783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:47.853642 19783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:47.854185 19783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:47.855082 19783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:47.855150 19783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:47.855221 19783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:47.855271 19783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:47.863857 19783 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:36111
I20260925 22:49:47.863914 19910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:36111 every 8 connection(s)
I20260925 22:49:47.864742 19783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:47.866945 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 19783
I20260925 22:49:47.867030 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:47.867367 806 external_mini_cluster.cc:1428] Setting key 660c8a439aad37acd400204c5fb06c78
I20260925 22:49:47.874593 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.866577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:34207 (local address 127.0.201.190:40179)
0925 22:49:47.866752 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.866758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.867969 (+ 1211us) server_negotiation.cc:407] Connection header received
0925 22:49:47.869684 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.869687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.869729 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.869793 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:47.870606 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.871103 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.872352 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.872587 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.872662 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.874034 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:47.874037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:47.874040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:47.874052 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:47.874080 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:47.874109 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:47.874178 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.874390 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.874453 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:47.875520 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.866959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:40179 (local address 127.0.201.131:34207)
0925 22:49:47.867793 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:47.867856 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:47.869501 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:47.869922 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:47.869928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:47.870012 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:47.870496 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.870508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.871203 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.871206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.872245 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.872255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.872817 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.872839 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.873022 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.873717 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:47.873719 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:47.873787 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:47.873870 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:47.874217 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:47.874228 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:47.874236 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:49:47.874387 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 22:49:47.874659 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":116,"threads_started":1}
I20260925 22:49:47.876748 19911 heartbeater.cc:347] Connected to a master server at 127.0.201.190:40179
I20260925 22:49:47.876959 19911 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:47.877420 19911 heartbeater.cc:511] Master 127.0.201.190:40179 requested a full tablet report, sending...
I20260925 22:49:47.878400 19461 ts_manager.cc:194] Registered new tserver with Master: 28634657630b4493ad878e4ac5008e76 (127.0.201.131:36111)
I20260925 22:49:47.878968 19461 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.131:34207
I20260925 22:49:47.881234 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:49:47.887105 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.883165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.0.201.190:40179)
0925 22:49:47.883302 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.883307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.883491 (+ 184us) server_negotiation.cc:407] Connection header received
0925 22:49:47.883635 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.883639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.883734 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.883810 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:47.884087 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.884607 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.885450 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.885685 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.885794 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.886601 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:47.886604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:47.886607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:47.886619 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:47.886641 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:47.886654 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:47.886784 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.886921 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.886958 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:49:47.889753 19461 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:47.892799 19461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:47.906495 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.901387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:45213 (local address 127.0.0.1:36496)
0925 22:49:47.901738 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:47.901787 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:47.901921 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:47.902714 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:47.902717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:47.902733 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:47.902837 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.902842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.903880 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.903886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.905091 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.905108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.905946 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.905949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.906057 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.906098 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 22:49:47.906213 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:49:47.906330 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":5,"thread_start_us":81,"threads_started":1}
I20260925 22:49:47.906512 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.901619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.0.201.130:45213)
0925 22:49:47.902277 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.902288 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.902309 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:49:47.902375 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.902379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.902537 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.902661 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:47.902970 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.903778 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.905217 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.905832 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.905941 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.906146 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.906244 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.906344 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":112,"threads_started":1}
I20260925 22:49:47.907032 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.900852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44827 (local address 127.0.0.1:39726)
0925 22:49:47.901929 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:47.901942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:47.902030 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:47.902566 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:47.902574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:47.902603 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:47.902793 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.902800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.904327 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.904329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.905596 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.905606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.906658 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.906664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.906755 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.906777 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:49:47.906821 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 22:49:47.906881 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":79,"thread_start_us":97,"threads_started":1}
I20260925 22:49:47.907287 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.900678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36111 (local address 127.0.0.1:49766)
0925 22:49:47.901737 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:47.901786 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:47.901936 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:47.902614 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:47.902616 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:47.902628 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:47.902793 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.902800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.904575 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.904578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.905716 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.905723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.906860 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.906863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.906968 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.906998 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:49:47.907089 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:49:47.907159 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":65,"thread_start_us":90,"threads_started":1}
I20260925 22:49:47.907382 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.900821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.0.201.131:36111)
0925 22:49:47.901324 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.901338 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.901870 (+ 532us) server_negotiation.cc:407] Connection header received
0925 22:49:47.902108 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.902116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.902326 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.902428 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:47.902919 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.904162 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.905833 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.906652 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.906752 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.906934 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.907119 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.907216 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":166,"threads_started":1}
I20260925 22:49:47.907787 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.901072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.0.201.129:44827)
0925 22:49:47.901459 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.901470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.902095 (+ 625us) server_negotiation.cc:407] Connection header received
0925 22:49:47.902178 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.902183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.902379 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.902546 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:47.903012 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.904162 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.905725 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.906508 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.907284 (+ 776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.907452 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.907511 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.907595 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":117,"threads_started":1}
I20260925 22:49:47.909979 19845 tablet_service.cc:1467] Processing CreateTablet for tablet f465a6e10a78454aac48b8b142fbe308 (DEFAULT_TABLE table=test-table [id=5ed39b3bbace4f2f86be55a10bd84ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:47.910141 19714 tablet_service.cc:1467] Processing CreateTablet for tablet f465a6e10a78454aac48b8b142fbe308 (DEFAULT_TABLE table=test-table [id=5ed39b3bbace4f2f86be55a10bd84ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:47.910140 19583 tablet_service.cc:1467] Processing CreateTablet for tablet f465a6e10a78454aac48b8b142fbe308 (DEFAULT_TABLE table=test-table [id=5ed39b3bbace4f2f86be55a10bd84ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:47.911039 19845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f465a6e10a78454aac48b8b142fbe308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:47.911121 19583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f465a6e10a78454aac48b8b142fbe308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:47.911121 19714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f465a6e10a78454aac48b8b142fbe308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:47.917174 19928 tablet_bootstrap.cc:492] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: Bootstrap starting.
I20260925 22:49:47.918625 19930 tablet_bootstrap.cc:492] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b: Bootstrap starting.
I20260925 22:49:47.919631 19928 tablet_bootstrap.cc:654] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:47.920042 19929 tablet_bootstrap.cc:492] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: Bootstrap starting.
I20260925 22:49:47.920583 19930 tablet_bootstrap.cc:654] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:47.920542 19928 log.cc:824] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:47.921372 19930 log.cc:824] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:47.922199 19929 tablet_bootstrap.cc:654] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:47.922811 19929 log.cc:824] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:47.923264 19928 tablet_bootstrap.cc:492] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: No bootstrap required, opened a new log
I20260925 22:49:47.923487 19928 ts_tablet_manager.cc:1397] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260925 22:49:47.923837 19930 tablet_bootstrap.cc:492] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b: No bootstrap required, opened a new log
I20260925 22:49:47.924036 19930 ts_tablet_manager.cc:1397] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:49:47.925243 19929 tablet_bootstrap.cc:492] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: No bootstrap required, opened a new log
I20260925 22:49:47.925410 19929 ts_tablet_manager.cc:1397] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 22:49:47.927846 19930 raft_consensus.cc:348] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 } } peers { permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 } } peers { permanent_uuid: "d9a71311a5b8451ba764732010644819" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45213 } }
I20260925 22:49:47.927846 19928 raft_consensus.cc:348] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 } } peers { permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 } } peers { permanent_uuid: "d9a71311a5b8451ba764732010644819" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45213 } }
I20260925 22:49:47.928118 19928 raft_consensus.cc:374] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:47.928118 19930 raft_consensus.cc:374] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:47.928177 19930 raft_consensus.cc:729] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba17e9134298492e954b3bc574f1d38b, State: Initialized, Role: FOLLOWER
I20260925 22:49:47.928185 19928 raft_consensus.cc:729] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28634657630b4493ad878e4ac5008e76, State: Initialized, Role: FOLLOWER
I20260925 22:49:47.928164 19929 raft_consensus.cc:348] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 } } peers { permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 } } peers { permanent_uuid: "d9a71311a5b8451ba764732010644819" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45213 } }
I20260925 22:49:47.928427 19929 raft_consensus.cc:374] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:47.928483 19929 raft_consensus.cc:729] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a71311a5b8451ba764732010644819, State: Initialized, Role: FOLLOWER
I20260925 22:49:47.928643 19930 consensus_queue.cc:260] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 } } peers { permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 } } peers { permanent_uuid: "d9a71311a5b8451ba764732010644819" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45213 } }
I20260925 22:49:47.928643 19928 consensus_queue.cc:260] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 } } peers { permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 } } peers { permanent_uuid: "d9a71311a5b8451ba764732010644819" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45213 } }
I20260925 22:49:47.928938 19929 consensus_queue.cc:260] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 } } peers { permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 } } peers { permanent_uuid: "d9a71311a5b8451ba764732010644819" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45213 } }
I20260925 22:49:47.929497 19649 heartbeater.cc:503] Master 127.0.201.190:40179 was elected leader, sending a full tablet report...
I20260925 22:49:47.929582 19911 heartbeater.cc:503] Master 127.0.201.190:40179 was elected leader, sending a full tablet report...
I20260925 22:49:47.929863 19929 ts_tablet_manager.cc:1428] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260925 22:49:47.930208 19780 heartbeater.cc:503] Master 127.0.201.190:40179 was elected leader, sending a full tablet report...
I20260925 22:49:47.930640 19930 ts_tablet_manager.cc:1428] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:49:47.930684 19928 ts_tablet_manager.cc:1428] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
W20260925 22:49:47.937023 19781 tablet.cc:2369] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:47.942080 19936 raft_consensus.cc:482] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:47.942226 19936 raft_consensus.cc:504] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 } } peers { permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 } } peers { permanent_uuid: "d9a71311a5b8451ba764732010644819" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45213 } }
I20260925 22:49:47.943202 19936 leader_election.cc:288] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28634657630b4493ad878e4ac5008e76 (127.0.201.131:36111), ba17e9134298492e954b3bc574f1d38b (127.0.201.129:44827)
I20260925 22:49:47.947264 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.943594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:39141 (local address 127.0.201.129:44827)
0925 22:49:47.943752 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.943757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.943771 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:49:47.943821 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.943823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.943881 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.943981 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:47.944276 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.944960 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.946158 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.946737 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.946875 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.946933 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.947068 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.947121 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:49:47.947265 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.943416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44827 (local address 127.0.201.130:39141)
0925 22:49:47.943548 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:47.943566 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:47.943657 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:47.943995 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:47.943998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:47.944015 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:47.944131 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.944136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.945063 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.945066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.946044 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.946054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.946840 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.946843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.946944 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.946971 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:49:47.947055 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:49:47.947124 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 22:49:47.947621 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.943416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:36111 (local address 127.0.201.130:57087)
0925 22:49:47.944086 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:47.944101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:47.944198 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:47.944451 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:47.944454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:47.944472 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:47.944611 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.944617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.945405 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.945408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.946434 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:47.946445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.947242 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.947245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:47.947358 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.947382 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:49:47.947465 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:49:47.947522 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":237,"thread_start_us":173,"threads_started":1}
I20260925 22:49:47.947738 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.943589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:57087 (local address 127.0.201.131:36111)
0925 22:49:47.943738 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.943744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.944196 (+ 452us) server_negotiation.cc:407] Connection header received
0925 22:49:47.944315 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.944318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.944366 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.944424 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:47.944709 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.945303 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.946574 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.947126 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.947231 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.947269 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.947503 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.947593 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:49:47.948071 19603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f465a6e10a78454aac48b8b142fbe308" candidate_uuid: "d9a71311a5b8451ba764732010644819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba17e9134298492e954b3bc574f1d38b" is_pre_election: true
I20260925 22:49:47.948258 19865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f465a6e10a78454aac48b8b142fbe308" candidate_uuid: "d9a71311a5b8451ba764732010644819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28634657630b4493ad878e4ac5008e76" is_pre_election: true
I20260925 22:49:47.948383 19603 raft_consensus.cc:2454] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a71311a5b8451ba764732010644819 in term 0.
I20260925 22:49:47.948508 19865 raft_consensus.cc:2454] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a71311a5b8451ba764732010644819 in term 0.
I20260925 22:49:47.948948 19669 leader_election.cc:302] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28634657630b4493ad878e4ac5008e76, ba17e9134298492e954b3bc574f1d38b, d9a71311a5b8451ba764732010644819; no voters:
I20260925 22:49:47.949288 19936 raft_consensus.cc:2781] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:47.949364 19936 raft_consensus.cc:482] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:47.949395 19936 raft_consensus.cc:3037] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:47.950523 19936 raft_consensus.cc:504] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 } } peers { permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 } } peers { permanent_uuid: "d9a71311a5b8451ba764732010644819" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45213 } }
I20260925 22:49:47.950887 19936 leader_election.cc:288] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [CANDIDATE]: Term 1 election: Requested vote from peers 28634657630b4493ad878e4ac5008e76 (127.0.201.131:36111), ba17e9134298492e954b3bc574f1d38b (127.0.201.129:44827)
I20260925 22:49:47.951164 19865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f465a6e10a78454aac48b8b142fbe308" candidate_uuid: "d9a71311a5b8451ba764732010644819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28634657630b4493ad878e4ac5008e76"
I20260925 22:49:47.951220 19603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f465a6e10a78454aac48b8b142fbe308" candidate_uuid: "d9a71311a5b8451ba764732010644819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba17e9134298492e954b3bc574f1d38b"
I20260925 22:49:47.951296 19865 raft_consensus.cc:3037] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:47.951385 19603 raft_consensus.cc:3037] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:47.952417 19865 raft_consensus.cc:2454] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a71311a5b8451ba764732010644819 in term 1.
I20260925 22:49:47.952441 19603 raft_consensus.cc:2454] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a71311a5b8451ba764732010644819 in term 1.
I20260925 22:49:47.952754 19668 leader_election.cc:302] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28634657630b4493ad878e4ac5008e76, ba17e9134298492e954b3bc574f1d38b, d9a71311a5b8451ba764732010644819; no voters:
I20260925 22:49:47.952992 19936 raft_consensus.cc:2781] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:47.953166 19936 raft_consensus.cc:686] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 1 LEADER]: Becoming Leader. State: Replica: d9a71311a5b8451ba764732010644819, State: Running, Role: LEADER
I20260925 22:49:47.953474 19936 consensus_queue.cc:237] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 } } peers { permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 } } peers { permanent_uuid: "d9a71311a5b8451ba764732010644819" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45213 } }
I20260925 22:49:47.956667 19459 catalog_manager.cc:5495] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9a71311a5b8451ba764732010644819 (127.0.201.130). New cstate: current_term: 1 leader_uuid: "d9a71311a5b8451ba764732010644819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9a71311a5b8451ba764732010644819" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 45213 } health_report { overall_health: HEALTHY } } }
I20260925 22:49:47.971170 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:47.967949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.0.201.130:45213)
0925 22:49:47.968081 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:47.968092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:47.968106 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:49:47.968213 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:47.968215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:47.968265 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:47.968347 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:47.968554 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.969049 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.969835 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:47.970042 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:47.970114 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:47.970389 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:47.970758 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:47.970866 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:47.970937 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:49:47.970966 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:49:47.977113 19603 raft_consensus.cc:1261] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b [term 1 FOLLOWER]: Refusing update from remote peer d9a71311a5b8451ba764732010644819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:47.977113 19865 raft_consensus.cc:1261] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76 [term 1 FOLLOWER]: Refusing update from remote peer d9a71311a5b8451ba764732010644819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:47.977810 19938 consensus_queue.cc:1035] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28634657630b4493ad878e4ac5008e76" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:47.977974 19936 consensus_queue.cc:1035] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba17e9134298492e954b3bc574f1d38b" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 44827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:47.985942 19942 mvcc.cc:205] Tried to move back new op lower bound from 7333382504346939392 to 7333382504260071424. Current Snapshot: MvccSnapshot[applied={T|T < 7333382504346939392}]
I20260925 22:49:47.991798 19943 mvcc.cc:205] Tried to move back new op lower bound from 7333382504346939392 to 7333382504260071424. Current Snapshot: MvccSnapshot[applied={T|T < 7333382504346939392 or (T in {7333382504346939392})}]
I20260925 22:49:47.993716 19460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:47.993881 19460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:47.996258 19460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b39c5535e849d89fd9b5ac2bb7667a: Sending DeleteTablet for 3 replicas of tablet f465a6e10a78454aac48b8b142fbe308
I20260925 22:49:47.997035 19583 tablet_service.cc:1514] Processing DeleteTablet for tablet f465a6e10a78454aac48b8b142fbe308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:47 UTC) from {username='slave'} at 127.0.0.1:39726
I20260925 22:49:47.997109 19845 tablet_service.cc:1514] Processing DeleteTablet for tablet f465a6e10a78454aac48b8b142fbe308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:47 UTC) from {username='slave'} at 127.0.0.1:49766
I20260925 22:49:47.997140 19714 tablet_service.cc:1514] Processing DeleteTablet for tablet f465a6e10a78454aac48b8b142fbe308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:47 UTC) from {username='slave'} at 127.0.0.1:36496
I20260925 22:49:47.997552 19951 tablet_replica.cc:330] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: stopping tablet replica
I20260925 22:49:47.997641 19950 tablet_replica.cc:330] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b: stopping tablet replica
I20260925 22:49:47.997848 19951 raft_consensus.cc:2229] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:47.997941 19950 raft_consensus.cc:2229] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:47.998193 19951 raft_consensus.cc:2258] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:47.998224 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 19521
I20260925 22:49:47.998276 19950 raft_consensus.cc:2258] T f465a6e10a78454aac48b8b142fbe308 P ba17e9134298492e954b3bc574f1d38b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:47.998520 19952 tablet_replica.cc:330] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: stopping tablet replica
I20260925 22:49:47.998814 19952 raft_consensus.cc:2229] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:47.999101 19952 raft_consensus.cc:2258] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:47.999814 19951 ts_tablet_manager.cc:1905] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:48.000593 19952 ts_tablet_manager.cc:1905] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:48.003063 19951 ts_tablet_manager.cc:1918] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:48.003176 19951 log.cc:1196] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/f465a6e10a78454aac48b8b142fbe308
I20260925 22:49:48.003187 19952 ts_tablet_manager.cc:1918] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:48.003283 19952 log.cc:1196] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/f465a6e10a78454aac48b8b142fbe308
I20260925 22:49:48.003562 19951 ts_tablet_manager.cc:1939] T f465a6e10a78454aac48b8b142fbe308 P d9a71311a5b8451ba764732010644819: Deleting consensus metadata
I20260925 22:49:48.003608 19952 ts_tablet_manager.cc:1939] T f465a6e10a78454aac48b8b142fbe308 P 28634657630b4493ad878e4ac5008e76: Deleting consensus metadata
I20260925 22:49:48.004737 19447 catalog_manager.cc:4841] TS 28634657630b4493ad878e4ac5008e76 (127.0.201.131:36111): tablet f465a6e10a78454aac48b8b142fbe308 (table test-table [id=5ed39b3bbace4f2f86be55a10bd84ca1]) successfully deleted
I20260925 22:49:48.004932 19447 catalog_manager.cc:4841] TS d9a71311a5b8451ba764732010644819 (127.0.201.130:45213): tablet f465a6e10a78454aac48b8b142fbe308 (table test-table [id=5ed39b3bbace4f2f86be55a10bd84ca1]) successfully deleted
W20260925 22:49:48.008343 19446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:44827 (error 108)
I20260925 22:49:48.008486 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 19652
I20260925 22:49:48.009902 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:48.009542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:44827 (local address 127.0.0.1:39736)
0925 22:49:48.009727 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:48.009800 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:44827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260925 22:49:48.010149 19446 catalog_manager.cc:4584] TS ba17e9134298492e954b3bc574f1d38b (127.0.201.129:44827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f465a6e10a78454aac48b8b142fbe308: Network error: Client connection negotiation failed: client connection to 127.0.201.129:44827: connect: Connection refused (error 111)
I20260925 22:49:48.014997 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 19783
I20260925 22:49:48.020164 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 19430
2026-09-25T22:49:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (964 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:48 dist-test-slave-vdtf krb5kdc[19956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:48 dist-test-slave-vdtf krb5kdc[19956](info): set up 2 sockets
Sep 25 22:49:48 dist-test-slave-vdtf krb5kdc[19956](info): commencing operation
krb5kdc: starting...
W20260925 22:49:49.071753 806 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:49:49 dist-test-slave-vdtf krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:49:49.191148 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:44365
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:36079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:44365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:49.298439 19969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:49.298723 19969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:49.298805 19969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:49.302837 19969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:49.302922 19969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:49.302959 19969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:49.303020 19969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:49.303079 19969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:49.308280 19969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:36079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:44365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:44365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:49.309345 19969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:49.310353 19969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:49.318658 19977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:49.318710 19975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:49.318710 19974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:49.319001 19969 server_base.cc:1034] running on GCE node
I20260925 22:49:49.319578 19969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:49.320680 19969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:49.321877 19969 hybrid_clock.cc:648] HybridClock initialized: now 1790376589321848 us; error 38 us; skew 500 ppm
Sep 25 22:49:49 dist-test-slave-vdtf krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:49.324697 19969 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:49:49.325826 19969 webserver.cc:460] Webserver started at http://127.0.201.190:41779/ using document root <none> and password file <none>
I20260925 22:49:49.326453 19969 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:49.326545 19969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:49.326754 19969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:49.328847 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "2f32614a1a9c4ed49ca624eae4747918"
format_stamp: "Formatted at 2026-09-25 22:49:49 on dist-test-slave-vdtf"
server_key: "b3e702906c3fa4370dddee0c9d83f873"
server_key_iv: "dd9e65d4ebbd212d8689068a6776537b"
server_key_version: "encryptionkey@0"
I20260925 22:49:49.329421 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "2f32614a1a9c4ed49ca624eae4747918"
format_stamp: "Formatted at 2026-09-25 22:49:49 on dist-test-slave-vdtf"
server_key: "b3e702906c3fa4370dddee0c9d83f873"
server_key_iv: "dd9e65d4ebbd212d8689068a6776537b"
server_key_version: "encryptionkey@0"
I20260925 22:49:49.333040 19969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:49.335333 19984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:49.336530 19969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:49.336668 19969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "2f32614a1a9c4ed49ca624eae4747918"
format_stamp: "Formatted at 2026-09-25 22:49:49 on dist-test-slave-vdtf"
server_key: "b3e702906c3fa4370dddee0c9d83f873"
server_key_iv: "dd9e65d4ebbd212d8689068a6776537b"
server_key_version: "encryptionkey@0"
I20260925 22:49:49.336778 19969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:49.349967 19969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:49.353231 19969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:49.353443 19969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:49.360251 19969 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:44365
I20260925 22:49:49.360284 20036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:44365 every 8 connection(s)
I20260925 22:49:49.361280 19969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:49.364012 20037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:49.366734 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 19969
I20260925 22:49:49.366894 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:49.367198 806 external_mini_cluster.cc:1428] Setting key 99cd28ba46158e1d27f7c426b7a9d259
I20260925 22:49:49.370191 20037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918: Bootstrap starting.
I20260925 22:49:49.372437 20037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:49.373171 20037 log.cc:824] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:49.375001 20037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918: No bootstrap required, opened a new log
Sep 25 22:49:49 dist-test-slave-vdtf krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:49.378762 20037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f32614a1a9c4ed49ca624eae4747918" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44365 } }
I20260925 22:49:49.379024 20037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:49.379108 20037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f32614a1a9c4ed49ca624eae4747918, State: Initialized, Role: FOLLOWER
I20260925 22:49:49.379621 20037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f32614a1a9c4ed49ca624eae4747918" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44365 } }
I20260925 22:49:49.379832 20037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:49.379913 20037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:49.380041 20037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:49.381073 20037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f32614a1a9c4ed49ca624eae4747918" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44365 } }
I20260925 22:49:49.381448 20037 leader_election.cc:302] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f32614a1a9c4ed49ca624eae4747918; no voters:
I20260925 22:49:49.381759 20037 leader_election.cc:288] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:49.381865 20042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:49.382077 20042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [term 1 LEADER]: Becoming Leader. State: Replica: 2f32614a1a9c4ed49ca624eae4747918, State: Running, Role: LEADER
I20260925 22:49:49.382376 20042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f32614a1a9c4ed49ca624eae4747918" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44365 } }
I20260925 22:49:49.382738 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:49.368689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.0.201.190:44365)
0925 22:49:49.369153 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:49.369169 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:49.369206 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:49:49.369822 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:49.369839 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:49.370185 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:49.370593 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:49.371435 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:49.372510 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:49.373441 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:49.373909 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:49.374079 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:49.376958 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:49.376963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:49.376976 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:49.377004 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:49.379374 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:49.379930 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:49.379936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:49.379958 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:49.380133 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:49.380432 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:49.380434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:49.380435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:49.380821 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:49.380992 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:49.381347 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 22:49:49.381685 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":118,"threads_started":1}
I20260925 22:49:49.383075 20037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:49.383538 20042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f32614a1a9c4ed49ca624eae4747918. Latest consensus state: current_term: 1 leader_uuid: "2f32614a1a9c4ed49ca624eae4747918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f32614a1a9c4ed49ca624eae4747918" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44365 } } }
I20260925 22:49:49.383646 20042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:49.384003 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f32614a1a9c4ed49ca624eae4747918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f32614a1a9c4ed49ca624eae4747918" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44365 } } }
I20260925 22:49:49.384107 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:49.384907 20046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:49.387997 20046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:49.393412 20046 catalog_manager.cc:1347] Generated new cluster ID: 20f4d67841d14a1694c64774bbd167c2
I20260925 22:49:49.393481 20046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:49.406111 20046 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:49.407325 20046 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:49.417444 20046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918: Generated new TSK 0
I20260925 22:49:49.418113 20046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.201.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
Entry for principal kudu/127.0.201.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab.
I20260925 22:49:49.452575 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44365
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:36079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:49.578497 20063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:49.578749 20063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:49.578791 20063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:49.582994 20063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:49.583045 20063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:49.583130 20063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:49:49.588934 20063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:36079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.129.keytab
--principal=kudu/127.0.201.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:49.589963 20063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:49.590965 20063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:49.600185 20068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:49.600248 20063 server_base.cc:1034] running on GCE node
W20260925 22:49:49.600255 20069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:49.600183 20071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:49.600881 20063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:49.601452 20063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:49.602617 20063 hybrid_clock.cc:648] HybridClock initialized: now 1790376589602596 us; error 27 us; skew 500 ppm
Sep 25 22:49:49 dist-test-slave-vdtf krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:49.605486 20063 init.cc:377] Logged in from keytab as kudu/127.0.201.129@KRBTEST.COM (short username kudu)
I20260925 22:49:49.606310 20063 webserver.cc:460] Webserver started at http://127.0.201.129:37255/ using document root <none> and password file <none>
I20260925 22:49:49.606868 20063 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:49.606928 20063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:49.607139 20063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:49.609299 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "2f9f73ce11a0400a848db6b55f5e33b9"
format_stamp: "Formatted at 2026-09-25 22:49:49 on dist-test-slave-vdtf"
server_key: "d60b279f9655cc4a3e64656080133c57"
server_key_iv: "9f03b8f42ebef537e7a4b0fd90f6bb07"
server_key_version: "encryptionkey@0"
I20260925 22:49:49.609869 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "2f9f73ce11a0400a848db6b55f5e33b9"
format_stamp: "Formatted at 2026-09-25 22:49:49 on dist-test-slave-vdtf"
server_key: "d60b279f9655cc4a3e64656080133c57"
server_key_iv: "9f03b8f42ebef537e7a4b0fd90f6bb07"
server_key_version: "encryptionkey@0"
I20260925 22:49:49.613613 20063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 22:49:49.616108 20078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:49.617444 20063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 22:49:49.617633 20063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "2f9f73ce11a0400a848db6b55f5e33b9"
format_stamp: "Formatted at 2026-09-25 22:49:49 on dist-test-slave-vdtf"
server_key: "d60b279f9655cc4a3e64656080133c57"
server_key_iv: "9f03b8f42ebef537e7a4b0fd90f6bb07"
server_key_version: "encryptionkey@0"
I20260925 22:49:49.617767 20063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:49.641776 20063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:49.644794 20063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:49.645032 20063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:49.645637 20063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:49.646469 20063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:49.646539 20063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:49.646608 20063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:49.646659 20063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:49.655790 20063 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:46621
I20260925 22:49:49.655823 20191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:46621 every 8 connection(s)
I20260925 22:49:49.656663 20063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:49.659351 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 20063
I20260925 22:49:49.659531 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:49.659911 806 external_mini_cluster.cc:1428] Setting key fc210db5bc7fe660144e4f4aaa39167d
Sep 25 22:49:49 dist-test-slave-vdtf krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.129@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.201.130@KRBTEST.COM; defaulting to no policy
I20260925 22:49:49.672235 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:49.658521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:54121 (local address 127.0.201.190:44365)
0925 22:49:49.658673 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:49.658678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:49.659691 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 22:49:49.661311 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:49.661314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:49.661363 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:49.661462 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:49.663066 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:49.664010 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:49.665212 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:49.665653 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:49.665775 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:49.668518 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:49.668525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:49.668530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:49.668563 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:49.670305 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:49.670766 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:49.670769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:49.670771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:49.670848 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:49.671171 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:49.671174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:49.671175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:49.671358 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:49.671434 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:49.672004 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 22:49:49.672108 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.130@KRBTEST.COM" created.
I20260925 22:49:49.673256 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:49.658982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44365 (local address 127.0.201.129:54121)
0925 22:49:49.659488 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:49.659557 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:49.661063 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:49.661668 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:49.661676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:49.662187 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:49.662896 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:49.662920 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:49.664167 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:49.664170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:49.665078 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:49.665088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:49.665875 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:49.665882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:49.666074 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:49.666770 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:49.666789 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:49.668347 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:49.670417 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:49.670422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:49.670434 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:49.670674 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:49.670934 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:49.670936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:49.670937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:49.671063 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:49.671450 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:49.671455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:49.671720 (+ 265us) client_negotiation.cc:769] Sending connection context
0925 22:49:49.671948 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 22:49:49.672217 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":134,"threads_started":1}
I20260925 22:49:49.674742 20192 heartbeater.cc:347] Connected to a master server at 127.0.201.190:44365
I20260925 22:49:49.675032 20192 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:49.675539 20192 heartbeater.cc:511] Master 127.0.201.190:44365 requested a full tablet report, sending...
I20260925 22:49:49.677116 20001 ts_manager.cc:194] Registered new tserver with Master: 2f9f73ce11a0400a848db6b55f5e33b9 (127.0.201.129:46621)
I20260925 22:49:49.678251 20001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.129@KRBTEST.COM'} at 127.0.201.129:54121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
Entry for principal kudu/127.0.201.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab.
I20260925 22:49:49.689049 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44365
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:36079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:49.798861 20197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:49.799108 20197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:49.799150 20197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:49.803426 20197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:49.803493 20197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:49.803557 20197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:49:49.809383 20197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:36079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.130.keytab
--principal=kudu/127.0.201.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:49.810369 20197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:49.811359 20197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:49.820706 20202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:49.820905 20205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:49.820706 20203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:49.821074 20197 server_base.cc:1034] running on GCE node
I20260925 22:49:49.821439 20197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:49.822104 20197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:49.823261 20197 hybrid_clock.cc:648] HybridClock initialized: now 1790376589823246 us; error 33 us; skew 500 ppm
Sep 25 22:49:49 dist-test-slave-vdtf krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:49.826073 20197 init.cc:377] Logged in from keytab as kudu/127.0.201.130@KRBTEST.COM (short username kudu)
I20260925 22:49:49.826920 20197 webserver.cc:460] Webserver started at http://127.0.201.130:36309/ using document root <none> and password file <none>
I20260925 22:49:49.827481 20197 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:49.827541 20197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:49.827827 20197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:49.829840 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "20abec4b47ab4f3dba9eb9f8916f4b12"
format_stamp: "Formatted at 2026-09-25 22:49:49 on dist-test-slave-vdtf"
server_key: "5ca448bfd534b7463aff7d4ad9d3c970"
server_key_iv: "b66ee99c26fec38cbecc83c0044fcd9e"
server_key_version: "encryptionkey@0"
I20260925 22:49:49.830401 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "20abec4b47ab4f3dba9eb9f8916f4b12"
format_stamp: "Formatted at 2026-09-25 22:49:49 on dist-test-slave-vdtf"
server_key: "5ca448bfd534b7463aff7d4ad9d3c970"
server_key_iv: "b66ee99c26fec38cbecc83c0044fcd9e"
server_key_version: "encryptionkey@0"
I20260925 22:49:49.833983 20197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:49.836688 20212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:49.838409 20197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:49.838548 20197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "20abec4b47ab4f3dba9eb9f8916f4b12"
format_stamp: "Formatted at 2026-09-25 22:49:49 on dist-test-slave-vdtf"
server_key: "5ca448bfd534b7463aff7d4ad9d3c970"
server_key_iv: "b66ee99c26fec38cbecc83c0044fcd9e"
server_key_version: "encryptionkey@0"
I20260925 22:49:49.838681 20197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:49.851351 20197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:49.854298 20197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:49.854519 20197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:49.855106 20197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:49.855994 20197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:49.856065 20197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:49.856130 20197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:49.856177 20197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:49.865185 20197 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:41989
I20260925 22:49:49.865216 20325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:41989 every 8 connection(s)
I20260925 22:49:49.866037 20197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
Sep 25 22:49:49 dist-test-slave-vdtf krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.130@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:49.875041 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 20197
I20260925 22:49:49.875195 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:49.875511 806 external_mini_cluster.cc:1428] Setting key 768e6295ff1e9d6c10d55760f3f9e35a
I20260925 22:49:49.879894 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:49.867937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:43157 (local address 127.0.201.190:44365)
0925 22:49:49.868076 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:49.868080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:49.868924 (+ 844us) server_negotiation.cc:407] Connection header received
0925 22:49:49.869821 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:49.869824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:49.869877 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:49.869985 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:49.871102 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:49.871638 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:49.872810 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:49.873044 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:49.873152 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:49.875777 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:49.875782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:49.875785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:49.875813 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:49.877576 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:49.878109 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:49.878115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:49.878120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:49.878218 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:49.878540 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:49.878546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:49.878550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:49.878813 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:49.878938 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:49.879590 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 22:49:49.879752 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:49:49.880839 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:49.868309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44365 (local address 127.0.201.130:43157)
0925 22:49:49.868758 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:49.868798 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:49.869620 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:49.870149 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:49.870156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:49.870499 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:49.870962 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:49.870975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:49.871813 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:49.871821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:49.872686 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:49.872694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:49.873204 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:49.873208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:49.873380 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:49.874004 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:49.874037 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:49.875577 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:49.877707 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:49.877712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:49.877724 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:49.878003 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:49.878313 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:49.878315 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:49.878316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:49.878447 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:49.878965 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:49.878973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:49.879311 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 22:49:49.879550 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 22:49:49.880002 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260925 22:49:49.882028 20326 heartbeater.cc:347] Connected to a master server at 127.0.201.190:44365
I20260925 22:49:49.882282 20326 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:49.882865 20326 heartbeater.cc:511] Master 127.0.201.190:44365 requested a full tablet report, sending...
I20260925 22:49:49.884063 20001 ts_manager.cc:194] Registered new tserver with Master: 20abec4b47ab4f3dba9eb9f8916f4b12 (127.0.201.130:41989)
I20260925 22:49:49.885004 20001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.130@KRBTEST.COM'} at 127.0.201.130:43157
WARNING: no policy specified for kudu/127.0.201.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
Entry for principal kudu/127.0.201.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab.
I20260925 22:49:49.906693 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44365
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:36079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:50.015995 20331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:50.016263 20331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:50.016337 20331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:50.020318 20331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:50.020396 20331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:50.020481 20331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:49:50.025754 20331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:36079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/krb5kdc/kudu_127.0.201.131.keytab
--principal=kudu/127.0.201.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:44365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:50.026760 20331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:50.027951 20331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:50.037251 20337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.037228 20336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.037222 20339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:50.037624 20331 server_base.cc:1034] running on GCE node
I20260925 22:49:50.038035 20331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:50.038620 20331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:50.039772 20331 hybrid_clock.cc:648] HybridClock initialized: now 1790376590039750 us; error 39 us; skew 500 ppm
Sep 25 22:49:50 dist-test-slave-vdtf krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:50.042668 20331 init.cc:377] Logged in from keytab as kudu/127.0.201.131@KRBTEST.COM (short username kudu)
I20260925 22:49:50.043592 20331 webserver.cc:460] Webserver started at http://127.0.201.131:36295/ using document root <none> and password file <none>
I20260925 22:49:50.044225 20331 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:50.044328 20331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:50.044551 20331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:50.046553 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "66387dca5f66448ebd94d8dddad8ea63"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "7d398b3190c6844ad736a690eac4aa71"
server_key_iv: "f302aa58ff90b1fd89ce0a86a3c4777e"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.047145 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "66387dca5f66448ebd94d8dddad8ea63"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "7d398b3190c6844ad736a690eac4aa71"
server_key_iv: "f302aa58ff90b1fd89ce0a86a3c4777e"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.050827 20331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:49:50.053447 20346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.054718 20331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:50.054876 20331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "66387dca5f66448ebd94d8dddad8ea63"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "7d398b3190c6844ad736a690eac4aa71"
server_key_iv: "f302aa58ff90b1fd89ce0a86a3c4777e"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.054989 20331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:50.069311 20331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:50.072043 20331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:50.072240 20331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:50.072798 20331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:50.073589 20331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:50.073660 20331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.073729 20331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:50.073782 20331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.082377 20331 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:44383
I20260925 22:49:50.082404 20459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:44383 every 8 connection(s)
I20260925 22:49:50.083309 20331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
Sep 25 22:49:50 dist-test-slave-vdtf krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.131@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:49:50.092944 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 20331
I20260925 22:49:50.093086 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:50.093425 806 external_mini_cluster.cc:1428] Setting key 5713a11bbaecae60fd1c8cbac0ee805b
I20260925 22:49:50.097364 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.085336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:43267 (local address 127.0.201.190:44365)
0925 22:49:50.085489 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.085492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.086413 (+ 921us) server_negotiation.cc:407] Connection header received
0925 22:49:50.087339 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.087342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.087400 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.087480 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:50.088721 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.089274 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.090435 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.090655 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.090770 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.093427 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:50.093433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:50.093439 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:50.093471 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:50.095230 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:50.095841 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:50.095844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:50.095846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:50.095927 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:50.096240 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:50.096242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:50.096244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:50.096430 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:50.096565 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.097136 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.097233 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:49:50.098191 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.085769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:44365 (local address 127.0.201.131:43267)
0925 22:49:50.086243 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.086286 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:50.087128 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:50.087645 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:50.087653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:50.088090 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:50.088570 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.088584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.089411 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.089418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.090271 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.090280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.090809 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.090813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.090988 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.091599 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:50.091617 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:50.093262 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:50.095342 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:50.095348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:50.095361 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:50.095733 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:50.096008 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:50.096010 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:50.096011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:50.096156 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:50.096556 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:50.096562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:50.096851 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 22:49:50.097053 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 22:49:50.097321 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
I20260925 22:49:50.099315 20460 heartbeater.cc:347] Connected to a master server at 127.0.201.190:44365
I20260925 22:49:50.099524 20460 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:50.100004 20460 heartbeater.cc:511] Master 127.0.201.190:44365 requested a full tablet report, sending...
I20260925 22:49:50.100925 20001 ts_manager.cc:194] Registered new tserver with Master: 66387dca5f66448ebd94d8dddad8ea63 (127.0.201.131:44383)
I20260925 22:49:50.101467 20001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.201.131@KRBTEST.COM'} at 127.0.201.131:43267
I20260925 22:49:50.107215 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:49:50.116176 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.109103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.0.201.190:44365)
0925 22:49:50.109219 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.109222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.109395 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:49:50.109482 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.109484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.109520 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.109581 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:50.110227 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.110740 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.111536 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.111823 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.111960 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.113139 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:50.113142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:50.113144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:50.113164 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:50.114586 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:50.114948 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:50.114951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:50.114952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:50.115029 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:50.115254 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:50.115256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:50.115257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:50.115508 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:50.115650 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.115894 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.116025 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:49:50.118587 20001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:50.121692 20001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:50.135888 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.129693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.0.201.130:41989)
0925 22:49:50.130441 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.130455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.130478 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:49:50.130556 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.130564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.130785 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.131298 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:50.131474 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.132650 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.134404 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.135186 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.135284 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.135394 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.135604 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.135729 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":160,"threads_started":1}
I20260925 22:49:50.135910 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.129553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:41989 (local address 127.0.0.1:45736)
0925 22:49:50.129914 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.129966 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:50.130118 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:50.130974 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:50.130979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:50.131003 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:50.131148 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.131155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.132802 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.132805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.134274 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.134285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.135296 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.135299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.135415 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.135453 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 22:49:50.135558 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 22:49:50.135740 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":121,"thread_start_us":73,"threads_started":1}
I20260925 22:49:50.136395 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.130622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.0.201.129:46621)
0925 22:49:50.131004 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.131024 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.131188 (+ 164us) server_negotiation.cc:407] Connection header received
0925 22:49:50.131354 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.131358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.131544 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.131738 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:50.131913 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.132619 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.134216 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.135066 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.135182 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.135343 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.136095 (+ 752us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.136198 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":105,"threads_started":1}
I20260925 22:49:50.136821 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.129932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:44383 (local address 127.0.0.1:60286)
0925 22:49:50.130537 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.130550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:50.130624 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:50.131462 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:50.131464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:50.131481 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:50.131576 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.131582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.133128 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.133131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.134279 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.134285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.136441 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.136444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.136539 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.136567 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:49:50.136625 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 22:49:50.136693 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":25,"thread_start_us":96,"threads_started":1}
I20260925 22:49:50.135910 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.130228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:46621 (local address 127.0.0.1:33888)
0925 22:49:50.131090 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.131101 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:50.131213 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:50.131708 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:50.131711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:50.131729 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:50.131816 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.131822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.132750 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.132756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.134094 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.134103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.135192 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.135195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.135366 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.135444 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 22:49:50.135558 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 22:49:50.135740 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":177,"thread_start_us":92,"threads_started":1}
I20260925 22:49:50.137490 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.130341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.0.201.131:44383)
0925 22:49:50.130849 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.130862 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.130884 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:49:50.131109 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.131114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.131314 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.131476 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:50.131948 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.132985 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.135452 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.136296 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.137114 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.137274 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.137315 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.137385 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":137,"threads_started":1}
I20260925 22:49:50.139281 20260 tablet_service.cc:1467] Processing CreateTablet for tablet a6bb727aed6548bb9558cf16db71a5e5 (DEFAULT_TABLE table=test-table [id=445bbe7fa2f14352bd3cf682984f6796]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:50.139492 20126 tablet_service.cc:1467] Processing CreateTablet for tablet a6bb727aed6548bb9558cf16db71a5e5 (DEFAULT_TABLE table=test-table [id=445bbe7fa2f14352bd3cf682984f6796]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:50.140079 20394 tablet_service.cc:1467] Processing CreateTablet for tablet a6bb727aed6548bb9558cf16db71a5e5 (DEFAULT_TABLE table=test-table [id=445bbe7fa2f14352bd3cf682984f6796]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:50.140489 20260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6bb727aed6548bb9558cf16db71a5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:50.140607 20126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6bb727aed6548bb9558cf16db71a5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:50.140952 20394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6bb727aed6548bb9558cf16db71a5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:50.146714 20477 tablet_bootstrap.cc:492] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9: Bootstrap starting.
I20260925 22:49:50.147492 20478 tablet_bootstrap.cc:492] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12: Bootstrap starting.
I20260925 22:49:50.148357 20479 tablet_bootstrap.cc:492] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: Bootstrap starting.
I20260925 22:49:50.149703 20477 tablet_bootstrap.cc:654] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:50.150120 20478 tablet_bootstrap.cc:654] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:50.150323 20479 tablet_bootstrap.cc:654] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:50.150997 20477 log.cc:824] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:50.150997 20478 log.cc:824] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:50.151321 20479 log.cc:824] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:50.152920 20477 tablet_bootstrap.cc:492] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9: No bootstrap required, opened a new log
I20260925 22:49:50.153053 20478 tablet_bootstrap.cc:492] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12: No bootstrap required, opened a new log
I20260925 22:49:50.153054 20479 tablet_bootstrap.cc:492] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: No bootstrap required, opened a new log
I20260925 22:49:50.153157 20477 ts_tablet_manager.cc:1397] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260925 22:49:50.153231 20478 ts_tablet_manager.cc:1397] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:49:50.153231 20479 ts_tablet_manager.cc:1397] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:49:50.156258 20477 raft_consensus.cc:348] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 } } peers { permanent_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46621 } } peers { permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 } }
I20260925 22:49:50.156495 20477 raft_consensus.cc:374] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:50.156558 20477 raft_consensus.cc:729] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f9f73ce11a0400a848db6b55f5e33b9, State: Initialized, Role: FOLLOWER
I20260925 22:49:50.157018 20477 consensus_queue.cc:260] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 } } peers { permanent_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46621 } } peers { permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 } }
I20260925 22:49:50.157033 20478 raft_consensus.cc:348] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 } } peers { permanent_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46621 } } peers { permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 } }
I20260925 22:49:50.157292 20478 raft_consensus.cc:374] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:50.157361 20478 raft_consensus.cc:729] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20abec4b47ab4f3dba9eb9f8916f4b12, State: Initialized, Role: FOLLOWER
I20260925 22:49:50.157325 20479 raft_consensus.cc:348] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 } } peers { permanent_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46621 } } peers { permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 } }
I20260925 22:49:50.157565 20479 raft_consensus.cc:374] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:50.157611 20479 raft_consensus.cc:729] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66387dca5f66448ebd94d8dddad8ea63, State: Initialized, Role: FOLLOWER
I20260925 22:49:50.157791 20478 consensus_queue.cc:260] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 } } peers { permanent_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46621 } } peers { permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 } }
I20260925 22:49:50.157933 20477 ts_tablet_manager.cc:1428] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260925 22:49:50.158020 20479 consensus_queue.cc:260] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 } } peers { permanent_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46621 } } peers { permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 } }
I20260925 22:49:50.158339 20192 heartbeater.cc:503] Master 127.0.201.190:44365 was elected leader, sending a full tablet report...
I20260925 22:49:50.158701 20478 ts_tablet_manager.cc:1428] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:49:50.158735 20460 heartbeater.cc:503] Master 127.0.201.190:44365 was elected leader, sending a full tablet report...
I20260925 22:49:50.158812 20479 ts_tablet_manager.cc:1428] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:49:50.159026 20326 heartbeater.cc:503] Master 127.0.201.190:44365 was elected leader, sending a full tablet report...
I20260925 22:49:50.187232 20483 raft_consensus.cc:482] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:50.187412 20483 raft_consensus.cc:504] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 } } peers { permanent_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46621 } } peers { permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 } }
I20260925 22:49:50.188549 20483 leader_election.cc:288] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20abec4b47ab4f3dba9eb9f8916f4b12 (127.0.201.130:41989), 66387dca5f66448ebd94d8dddad8ea63 (127.0.201.131:44383)
I20260925 22:49:50.193946 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.188862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:59229 (local address 127.0.201.130:41989)
0925 22:49:50.189056 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.189061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.189154 (+ 93us) server_negotiation.cc:407] Connection header received
0925 22:49:50.189321 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.189324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.189379 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.189464 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:50.189870 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.190671 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.192683 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.193298 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.193404 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.193450 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.193746 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.193821 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 22:49:50.193946 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.188973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:43111 (local address 127.0.201.131:44383)
0925 22:49:50.189086 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.189090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.189260 (+ 170us) server_negotiation.cc:407] Connection header received
0925 22:49:50.189334 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.189336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.189380 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.189463 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:50.189870 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.190671 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.192625 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.193267 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.193399 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.193450 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.193746 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.193820 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:49:50.194006 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.188804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:44383 (local address 127.0.201.129:43111)
0925 22:49:50.189159 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.189170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:50.189257 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:50.189494 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:50.189497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:50.189538 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:50.189739 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.189745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.190786 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.190790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.192437 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.192449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.193417 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.193422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.193591 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.193635 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 22:49:50.193712 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:49:50.193823 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":283,"spinlock_wait_cycles":6016,"thread_start_us":105,"threads_started":1}
I20260925 22:49:50.194010 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.188709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:41989 (local address 127.0.201.129:59229)
0925 22:49:50.189052 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.189069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:50.189186 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:50.189494 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:50.189497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:50.189537 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:50.189738 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.189745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.190786 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.190790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.192437 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.192449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.193417 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.193422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.193588 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.193631 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 22:49:50.193712 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:49:50.193827 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":262,"spinlock_wait_cycles":35840,"thread_start_us":111,"threads_started":1}
I20260925 22:49:50.194681 20280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bb727aed6548bb9558cf16db71a5e5" candidate_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" is_pre_election: true
I20260925 22:49:50.194743 20414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bb727aed6548bb9558cf16db71a5e5" candidate_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66387dca5f66448ebd94d8dddad8ea63" is_pre_election: true
I20260925 22:49:50.195016 20414 raft_consensus.cc:2454] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f9f73ce11a0400a848db6b55f5e33b9 in term 0.
I20260925 22:49:50.195016 20280 raft_consensus.cc:2454] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f9f73ce11a0400a848db6b55f5e33b9 in term 0.
I20260925 22:49:50.195552 20079 leader_election.cc:302] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20abec4b47ab4f3dba9eb9f8916f4b12, 2f9f73ce11a0400a848db6b55f5e33b9, 66387dca5f66448ebd94d8dddad8ea63; no voters:
I20260925 22:49:50.195834 20483 raft_consensus.cc:2781] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:50.195920 20483 raft_consensus.cc:482] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:50.195966 20483 raft_consensus.cc:3037] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:50.197009 20483 raft_consensus.cc:504] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 } } peers { permanent_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46621 } } peers { permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 } }
I20260925 22:49:50.197407 20483 leader_election.cc:288] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [CANDIDATE]: Term 1 election: Requested vote from peers 20abec4b47ab4f3dba9eb9f8916f4b12 (127.0.201.130:41989), 66387dca5f66448ebd94d8dddad8ea63 (127.0.201.131:44383)
I20260925 22:49:50.197731 20280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bb727aed6548bb9558cf16db71a5e5" candidate_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12"
I20260925 22:49:50.197808 20414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bb727aed6548bb9558cf16db71a5e5" candidate_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66387dca5f66448ebd94d8dddad8ea63"
I20260925 22:49:50.197857 20280 raft_consensus.cc:3037] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:50.197978 20414 raft_consensus.cc:3037] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:50.198925 20280 raft_consensus.cc:2454] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f9f73ce11a0400a848db6b55f5e33b9 in term 1.
I20260925 22:49:50.199064 20414 raft_consensus.cc:2454] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f9f73ce11a0400a848db6b55f5e33b9 in term 1.
I20260925 22:49:50.199232 20079 leader_election.cc:302] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20abec4b47ab4f3dba9eb9f8916f4b12, 2f9f73ce11a0400a848db6b55f5e33b9; no voters:
I20260925 22:49:50.199406 20483 raft_consensus.cc:2781] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:50.199635 20483 raft_consensus.cc:686] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f9f73ce11a0400a848db6b55f5e33b9, State: Running, Role: LEADER
I20260925 22:49:50.200016 20483 consensus_queue.cc:237] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 } } peers { permanent_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46621 } } peers { permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 } }
I20260925 22:49:50.202987 20000 catalog_manager.cc:5495] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f9f73ce11a0400a848db6b55f5e33b9 (127.0.201.129). New cstate: current_term: 1 leader_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f9f73ce11a0400a848db6b55f5e33b9" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 46621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 } health_report { overall_health: UNKNOWN } } }
I20260925 22:49:50.216071 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.212913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.0.201.129:46621)
0925 22:49:50.213034 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.213038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.213050 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:49:50.213110 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.213112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.213159 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.213245 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:50.213462 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.213943 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.214744 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.214962 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.215042 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.215299 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:50.215762 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:50.215853 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.215918 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.215969 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:50.221699 20414 raft_consensus.cc:1261] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63 [term 1 FOLLOWER]: Refusing update from remote peer 2f9f73ce11a0400a848db6b55f5e33b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:50.221751 20280 raft_consensus.cc:1261] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12 [term 1 FOLLOWER]: Refusing update from remote peer 2f9f73ce11a0400a848db6b55f5e33b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:50.222404 20483 consensus_queue.cc:1035] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20abec4b47ab4f3dba9eb9f8916f4b12" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:50.222620 20488 consensus_queue.cc:1035] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66387dca5f66448ebd94d8dddad8ea63" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:50.229566 20491 mvcc.cc:205] Tried to move back new op lower bound from 7333382513541357568 to 7333382513461821440. Current Snapshot: MvccSnapshot[applied={T|T < 7333382513541357568}]
I20260925 22:49:50.230041 20493 mvcc.cc:205] Tried to move back new op lower bound from 7333382513541357568 to 7333382513461821440. Current Snapshot: MvccSnapshot[applied={T|T < 7333382513541357568}]
I20260925 22:49:50.231825 20492 mvcc.cc:205] Tried to move back new op lower bound from 7333382513541357568 to 7333382513461821440. Current Snapshot: MvccSnapshot[applied={T|T < 7333382513541357568}]
I20260925 22:49:50.238919 20000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:50.239110 20000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:50.241513 20000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f32614a1a9c4ed49ca624eae4747918: Sending DeleteTablet for 3 replicas of tablet a6bb727aed6548bb9558cf16db71a5e5
I20260925 22:49:50.242307 20260 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bb727aed6548bb9558cf16db71a5e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:50 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:45736
I20260925 22:49:50.242339 20394 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bb727aed6548bb9558cf16db71a5e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:50 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:60286
I20260925 22:49:50.242348 20126 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bb727aed6548bb9558cf16db71a5e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:50 UTC) from {username='kudu', principal='kudu/127.0.201.190@KRBTEST.COM'} at 127.0.0.1:33888
I20260925 22:49:50.242929 20502 tablet_replica.cc:330] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: stopping tablet replica
I20260925 22:49:50.243177 20501 tablet_replica.cc:330] T a6bb727aed6548bb9558cf16db71a5e5 P 2f9f73ce11a0400a848db6b55f5e33b9: stopping tablet replica
I20260925 22:49:50.243387 20502 raft_consensus.cc:2229] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:50.243546 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 20063
I20260925 22:49:50.245034 20502 raft_consensus.cc:2258] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:50.246402 20502 ts_tablet_manager.cc:1905] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:50.242941 20500 tablet_replica.cc:330] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12: stopping tablet replica
I20260925 22:49:50.248100 20500 raft_consensus.cc:2229] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:50.248483 20500 raft_consensus.cc:2258] T a6bb727aed6548bb9558cf16db71a5e5 P 20abec4b47ab4f3dba9eb9f8916f4b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:50.248932 20502 ts_tablet_manager.cc:1918] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:50.249037 20502 log.cc:1196] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/a6bb727aed6548bb9558cf16db71a5e5
I20260925 22:49:50.249384 20502 ts_tablet_manager.cc:1939] T a6bb727aed6548bb9558cf16db71a5e5 P 66387dca5f66448ebd94d8dddad8ea63: Deleting consensus metadata
W20260925 22:49:50.250029 19985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:46621 (error 108)
I20260925 22:49:50.250069 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 20197
I20260925 22:49:50.251911 19986 catalog_manager.cc:4841] TS 66387dca5f66448ebd94d8dddad8ea63 (127.0.201.131:44383): tablet a6bb727aed6548bb9558cf16db71a5e5 (table test-table [id=445bbe7fa2f14352bd3cf682984f6796]) successfully deleted
I20260925 22:49:50.252116 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.251635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:46621 (local address 127.0.0.1:33910)
0925 22:49:50.251926 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.251995 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
W20260925 22:49:50.252269 19985 catalog_manager.cc:4584] TS 2f9f73ce11a0400a848db6b55f5e33b9 (127.0.201.129:46621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6bb727aed6548bb9558cf16db71a5e5: Network error: Client connection negotiation failed: client connection to 127.0.201.129:46621: connect: Connection refused (error 111)
I20260925 22:49:50.255829 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 20331
I20260925 22:49:50.256234 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.255897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:41989 (local address 127.0.0.1:45744)
0925 22:49:50.256095 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.256149 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.130:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260925 22:49:50.261871 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 19969
2026-09-25T22:49:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2242 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T22:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:50Z Disabled control of system clock
I20260925 22:49:50.291996 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:34777
--webserver_interface=127.0.201.190
--webserver_port=0
--builtin_ntp_servers=127.0.201.148:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:34777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:49:50.401238 20507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:50.401518 20507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:50.401554 20507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:50.405593 20507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:50.405658 20507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:50.405673 20507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:50.405689 20507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:50.405704 20507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:50.410652 20507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:34777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:50.411839 20507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:50.413300 20507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:49:50.422003 20507 server_base.cc:1034] running on GCE node
W20260925 22:49:50.421962 20515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.421926 20512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.421926 20513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:50.422717 20507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:50.423830 20507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:50.424999 20507 hybrid_clock.cc:648] HybridClock initialized: now 1790376590424988 us; error 24 us; skew 500 ppm
I20260925 22:49:50.426817 20507 webserver.cc:460] Webserver started at http://127.0.201.190:42241/ using document root <none> and password file <none>
I20260925 22:49:50.427470 20507 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:50.427531 20507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:50.427805 20507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:50.429873 20507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "9555102add054b0ca1ad7c93ee9669b5"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "452f31c78d0eeef97f6f00b57faefc4a"
server_key_iv: "65e6f0eff431eb6e61051c051070aae8"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.430442 20507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "9555102add054b0ca1ad7c93ee9669b5"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "452f31c78d0eeef97f6f00b57faefc4a"
server_key_iv: "65e6f0eff431eb6e61051c051070aae8"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.434106 20507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:50.436618 20521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.438067 20507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:50.438216 20507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "9555102add054b0ca1ad7c93ee9669b5"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "452f31c78d0eeef97f6f00b57faefc4a"
server_key_iv: "65e6f0eff431eb6e61051c051070aae8"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.438336 20507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:50.466126 20507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:50.466871 20507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:50.467084 20507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:50.474140 20507 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:34777
I20260925 22:49:50.474152 20573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:34777 every 8 connection(s)
I20260925 22:49:50.475047 20507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:50.477838 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 20507
I20260925 22:49:50.477990 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:50.478049 20574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:50.478338 806 external_mini_cluster.cc:1428] Setting key 6f051beda724c4d355452a9f5584d660
I20260925 22:49:50.484629 20574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5: Bootstrap starting.
I20260925 22:49:50.487108 20574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:50.487658 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.479627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.0.201.190:34777)
0925 22:49:50.480249 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.480264 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.480299 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:49:50.481156 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.481175 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.481544 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.481816 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:50.482150 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.483064 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.484340 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.484934 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.485054 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.485915 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:50.485920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:50.485932 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:50.486331 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:50.486386 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:50.486405 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:50.486479 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.486638 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.486935 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":124,"threads_started":1}
I20260925 22:49:50.487943 20574 log.cc:824] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:50.490016 20574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5: No bootstrap required, opened a new log
I20260925 22:49:50.493422 20574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555102add054b0ca1ad7c93ee9669b5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34777 } }
I20260925 22:49:50.493640 20574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:50.493716 20574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9555102add054b0ca1ad7c93ee9669b5, State: Initialized, Role: FOLLOWER
I20260925 22:49:50.494100 20574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555102add054b0ca1ad7c93ee9669b5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34777 } }
I20260925 22:49:50.494238 20574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:50.494318 20574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:50.494437 20574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:50.495224 20574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555102add054b0ca1ad7c93ee9669b5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34777 } }
I20260925 22:49:50.495540 20574 leader_election.cc:302] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9555102add054b0ca1ad7c93ee9669b5; no voters:
I20260925 22:49:50.495842 20574 leader_election.cc:288] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:50.495950 20579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:50.496212 20579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9555102add054b0ca1ad7c93ee9669b5, State: Running, Role: LEADER
I20260925 22:49:50.496483 20579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555102add054b0ca1ad7c93ee9669b5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34777 } }
I20260925 22:49:50.496932 20574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:50.498106 20581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9555102add054b0ca1ad7c93ee9669b5. Latest consensus state: current_term: 1 leader_uuid: "9555102add054b0ca1ad7c93ee9669b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555102add054b0ca1ad7c93ee9669b5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34777 } } }
I20260925 22:49:50.498138 20580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9555102add054b0ca1ad7c93ee9669b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555102add054b0ca1ad7c93ee9669b5" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34777 } } }
I20260925 22:49:50.498221 20581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:50.498245 20580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:50.498769 20588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:50.501487 20588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:50.506430 20588 catalog_manager.cc:1347] Generated new cluster ID: 22dc7907f8d24eda966916716aedc1dc
I20260925 22:49:50.506493 20588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:50.514093 20588 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:50.514927 20588 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:50.529999 20588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5: Generated new TSK 0
I20260925 22:49:50.530754 20588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:50.535986 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.129:0
--local_ip_for_outbound_sockets=127.0.201.129
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:34777
--builtin_ntp_servers=127.0.201.148:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:49:50.646183 20598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:50.646417 20598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:50.646452 20598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:50.650831 20598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:50.650885 20598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:50.650940 20598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.129
I20260925 22:49:50.655622 20598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.201.129
--webserver_port=0
--tserver_master_addrs=127.0.201.190:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.129
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:50.656625 20598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:50.657935 20598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:50.667315 20604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.667325 20603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:50.667418 20598 server_base.cc:1034] running on GCE node
W20260925 22:49:50.667325 20606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:50.667950 20598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:50.668537 20598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:50.669690 20598 hybrid_clock.cc:648] HybridClock initialized: now 1790376590669645 us; error 58 us; skew 500 ppm
I20260925 22:49:50.671353 20598 webserver.cc:460] Webserver started at http://127.0.201.129:33297/ using document root <none> and password file <none>
I20260925 22:49:50.671933 20598 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:50.671993 20598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:50.672147 20598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:50.674046 20598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/data/instance:
uuid: "63a504a703da430ebee5e9862cef8783"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "019a79d709ad6b8aa26605171f43d911"
server_key_iv: "2904a2f2e1d4c31cad29d4eb5f7a0c1f"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.674525 20598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/wal/instance:
uuid: "63a504a703da430ebee5e9862cef8783"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "019a79d709ad6b8aa26605171f43d911"
server_key_iv: "2904a2f2e1d4c31cad29d4eb5f7a0c1f"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.678022 20598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:50.680280 20612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.681489 20598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:50.681582 20598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/wal
uuid: "63a504a703da430ebee5e9862cef8783"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "019a79d709ad6b8aa26605171f43d911"
server_key_iv: "2904a2f2e1d4c31cad29d4eb5f7a0c1f"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.681663 20598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:50.695779 20598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:50.696363 20598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:50.696499 20598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:50.696993 20598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:50.697929 20598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:50.697973 20598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.698083 20598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:50.698110 20598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.706809 20598 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.129:39065
I20260925 22:49:50.706844 20725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.129:39065 every 8 connection(s)
I20260925 22:49:50.707729 20598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:50.712019 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 20598
I20260925 22:49:50.712152 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:50.712486 806 external_mini_cluster.cc:1428] Setting key 2bb053fd238741a0884c2f3d3569f33b
I20260925 22:49:50.714393 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.130:0
--local_ip_for_outbound_sockets=127.0.201.130
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:34777
--builtin_ntp_servers=127.0.201.148:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:49:50.717483 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.709579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:56935 (local address 127.0.201.190:34777)
0925 22:49:50.709731 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.709735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.710800 (+ 1065us) server_negotiation.cc:407] Connection header received
0925 22:49:50.712155 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.712161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.712224 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.712335 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:50.713209 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.713847 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.715065 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.715479 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.715603 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.716851 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:50.716854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:50.716856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:50.716867 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:50.716885 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:50.716893 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:50.716998 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.717184 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.717272 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:49:50.718143 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.710034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:34777 (local address 127.0.201.129:56935)
0925 22:49:50.710647 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.710686 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:50.711936 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:50.712483 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:50.712490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:50.712615 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:50.713076 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.713088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.713971 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.713973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.714941 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.714950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.715643 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.715650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.715920 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.716543 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:50.716547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:50.716601 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:50.716681 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:50.717008 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:50.717015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:50.717019 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:49:50.717148 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 22:49:50.717355 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260925 22:49:50.720072 20726 heartbeater.cc:347] Connected to a master server at 127.0.201.190:34777
I20260925 22:49:50.720330 20726 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:50.720958 20726 heartbeater.cc:511] Master 127.0.201.190:34777 requested a full tablet report, sending...
I20260925 22:49:50.722466 20538 ts_manager.cc:194] Registered new tserver with Master: 63a504a703da430ebee5e9862cef8783 (127.0.201.129:39065)
I20260925 22:49:50.723567 20538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.129:56935
W20260925 22:49:50.827275 20729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:50.827550 20729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:50.827621 20729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:50.831647 20729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:50.831734 20729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:50.831825 20729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.130
I20260925 22:49:50.837136 20729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.201.130
--webserver_port=0
--tserver_master_addrs=127.0.201.190:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.130
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:50.838191 20729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:50.839511 20729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:50.849268 20737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.849291 20734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.849284 20735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:50.849424 20729 server_base.cc:1034] running on GCE node
I20260925 22:49:50.849887 20729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:50.850476 20729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:50.851652 20729 hybrid_clock.cc:648] HybridClock initialized: now 1790376590851607 us; error 57 us; skew 500 ppm
I20260925 22:49:50.853356 20729 webserver.cc:460] Webserver started at http://127.0.201.130:34749/ using document root <none> and password file <none>
I20260925 22:49:50.853924 20729 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:50.854012 20729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:50.854244 20729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:50.856256 20729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/data/instance:
uuid: "6890e786a8b34582bf2220ba5afc0b0f"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "77e491f54c68f11c0d32f2d1b59d1289"
server_key_iv: "0d23015f32afd8f7f72b761b916ee537"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.856794 20729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/wal/instance:
uuid: "6890e786a8b34582bf2220ba5afc0b0f"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "77e491f54c68f11c0d32f2d1b59d1289"
server_key_iv: "0d23015f32afd8f7f72b761b916ee537"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.860229 20729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:50.862462 20743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.863602 20729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:50.863754 20729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/wal
uuid: "6890e786a8b34582bf2220ba5afc0b0f"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-vdtf"
server_key: "77e491f54c68f11c0d32f2d1b59d1289"
server_key_iv: "0d23015f32afd8f7f72b761b916ee537"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.863903 20729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:50.899164 20729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:50.900159 20729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:50.900373 20729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:50.901173 20729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:50.902500 20729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:50.902565 20729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.902624 20729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:50.902652 20729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.914156 20729 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.130:39531
I20260925 22:49:50.914247 20856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.130:39531 every 8 connection(s)
I20260925 22:49:50.915326 20729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:50.921473 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 20729
I20260925 22:49:50.921597 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:50.921911 806 external_mini_cluster.cc:1428] Setting key 5dcebbdf6642db362718d8fb9fb738a3
I20260925 22:49:50.923916 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.201.131:0
--local_ip_for_outbound_sockets=127.0.201.131
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:34777
--builtin_ntp_servers=127.0.201.148:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:49:50.928109 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.919105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.130:46569 (local address 127.0.201.190:34777)
0925 22:49:50.919252 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.919257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.919274 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:49:50.921119 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.921122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.921176 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.921273 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:50.922671 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.923410 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.925375 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.925791 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.925899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.927437 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:50.927442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:50.927448 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:50.927461 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:50.927485 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:50.927497 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:50.927550 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.927846 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.927925 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:49:50.929774 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.917948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:34777 (local address 127.0.201.130:46569)
0925 22:49:50.918806 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.918864 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:50.920658 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:50.921593 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:50.921605 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:50.921730 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:50.922487 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.922513 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.923591 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.923596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.925218 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.925231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.926119 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.926124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.926365 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.927078 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:50.927081 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:50.927149 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:50.927241 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:50.927626 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:50.927636 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:50.927644 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:49:50.928320 (+ 676us) client_negotiation.cc:240] Negotiation successful
0925 22:49:50.928650 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":157,"threads_started":1}
I20260925 22:49:50.931057 20857 heartbeater.cc:347] Connected to a master server at 127.0.201.190:34777
I20260925 22:49:50.931316 20857 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:50.932040 20857 heartbeater.cc:511] Master 127.0.201.190:34777 requested a full tablet report, sending...
I20260925 22:49:50.933312 20538 ts_manager.cc:194] Registered new tserver with Master: 6890e786a8b34582bf2220ba5afc0b0f (127.0.201.130:39531)
I20260925 22:49:50.934197 20538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.130:46569
W20260925 22:49:51.036617 20860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:51.036890 20860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:51.036957 20860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:51.040930 20860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:51.041002 20860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:51.041085 20860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.201.131
I20260925 22:49:51.045914 20860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.201.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.201.131
--webserver_port=0
--tserver_master_addrs=127.0.201.190:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.201.131
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:51.046926 20860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:51.048290 20860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:51.057293 20868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:51.057334 20865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:51.057304 20866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:51.057678 20860 server_base.cc:1034] running on GCE node
I20260925 22:49:51.058035 20860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:51.058619 20860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:51.059741 20860 hybrid_clock.cc:648] HybridClock initialized: now 1790376591059728 us; error 25 us; skew 500 ppm
I20260925 22:49:51.069931 20860 webserver.cc:460] Webserver started at http://127.0.201.131:35079/ using document root <none> and password file <none>
I20260925 22:49:51.070554 20860 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:51.070616 20860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:51.070856 20860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:51.073027 20860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/data/instance:
uuid: "907deffd30eb4b8f80e1141836ea4bb8"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-vdtf"
server_key: "c8cd825d60ba25fbfc112b1c8958cb6e"
server_key_iv: "72cb36e2a6190675deec748a40fc62a6"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.073580 20860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/wal/instance:
uuid: "907deffd30eb4b8f80e1141836ea4bb8"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-vdtf"
server_key: "c8cd825d60ba25fbfc112b1c8958cb6e"
server_key_iv: "72cb36e2a6190675deec748a40fc62a6"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.077210 20860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 22:49:51.079497 20874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:51.080754 20860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:51.080900 20860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/wal
uuid: "907deffd30eb4b8f80e1141836ea4bb8"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-vdtf"
server_key: "c8cd825d60ba25fbfc112b1c8958cb6e"
server_key_iv: "72cb36e2a6190675deec748a40fc62a6"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.081014 20860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:51.104328 20860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:51.105057 20860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:51.105262 20860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:51.105885 20860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:51.106832 20860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:51.106909 20860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:51.106980 20860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:51.107030 20860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:51.115739 20860 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.131:37177
I20260925 22:49:51.115747 20987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.131:37177 every 8 connection(s)
I20260925 22:49:51.116642 20860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:51.120545 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 20860
I20260925 22:49:51.120666 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:51.120959 806 external_mini_cluster.cc:1428] Setting key e2e7a8774a900fd1d63b0136a372e144
I20260925 22:49:51.126034 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.118453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.131:48251 (local address 127.0.201.190:34777)
0925 22:49:51.118597 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.118602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.119657 (+ 1055us) server_negotiation.cc:407] Connection header received
0925 22:49:51.121177 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.121181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.121238 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.121335 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:51.122418 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.123121 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.124304 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.124516 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.124595 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.125588 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:51.125590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:51.125593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:51.125605 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:51.125624 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:51.125632 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:51.125691 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.125879 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.125928 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:51.126878 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.118903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:34777 (local address 127.0.201.131:48251)
0925 22:49:51.119517 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.119556 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.120899 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.121560 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.121567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.121650 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:51.122307 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.122324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.123255 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.123257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.124202 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.124211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.124644 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.124646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.124789 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.125338 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:51.125339 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:51.125384 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:51.125444 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:51.125726 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:51.125733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:51.125737 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.125852 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.126068 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260925 22:49:51.127938 20988 heartbeater.cc:347] Connected to a master server at 127.0.201.190:34777
I20260925 22:49:51.128165 20988 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:51.128652 20988 heartbeater.cc:511] Master 127.0.201.190:34777 requested a full tablet report, sending...
I20260925 22:49:51.129606 20538 ts_manager.cc:194] Registered new tserver with Master: 907deffd30eb4b8f80e1141836ea4bb8 (127.0.201.131:37177)
I20260925 22:49:51.130125 20538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.201.131:48251
I20260925 22:49:51.134680 806 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:49:51.140426 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.136567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.0.201.190:34777)
0925 22:49:51.136730 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.136735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.136978 (+ 243us) server_negotiation.cc:407] Connection header received
0925 22:49:51.137132 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.137135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.137181 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.137261 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:51.137531 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.138038 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.138889 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.139105 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.139213 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.139999 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:51.140002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:51.140005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:51.140023 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:51.140040 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:51.140048 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:51.140123 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.140241 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.140291 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:49:51.143183 20538 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:51.146174 20538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:51.159826 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.154294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.0.201.129:39065)
0925 22:49:51.154719 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.154732 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.155133 (+ 401us) server_negotiation.cc:407] Connection header received
0925 22:49:51.155436 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.155443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.155665 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.155821 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:51.156470 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.157202 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.158604 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.159143 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.159267 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.159472 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.159555 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.159643 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":101,"threads_started":1}
I20260925 22:49:51.159824 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.154144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:39065 (local address 127.0.0.1:46620)
0925 22:49:51.154992 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.155045 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.155246 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.156025 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.156031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.156068 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:51.156341 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.156351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.157317 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.157322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.158480 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.158489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.159267 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.159271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.159374 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.159411 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.159527 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.159643 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":95,"spinlock_wait_cycles":5376,"thread_start_us":70,"threads_started":1}
I20260925 22:49:51.160952 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.154548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:37177 (local address 127.0.0.1:57836)
0925 22:49:51.155473 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.155486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.155556 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.156022 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.156027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.156059 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:51.156260 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.156268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.157776 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.157779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.159266 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.159277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.160377 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.160381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.160472 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.160631 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.160710 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.160779 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":186,"spinlock_wait_cycles":3456,"thread_start_us":91,"threads_started":1}
I20260925 22:49:51.161633 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.154319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:39531 (local address 127.0.0.1:50222)
0925 22:49:51.155446 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.155460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.155625 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.156177 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.156179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.156196 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:51.156372 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.156378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.157776 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.157779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.159266 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.159277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.161355 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.161357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.161430 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.161445 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.161499 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.161547 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"mutex_wait_us":181,"thread_start_us":96,"threads_started":1}
I20260925 22:49:51.161818 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.154886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.0.201.130:39531)
0925 22:49:51.155311 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.155324 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.155548 (+ 224us) server_negotiation.cc:407] Connection header received
0925 22:49:51.155773 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.155778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.155963 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.156102 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:51.156527 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.157654 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.159412 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.160230 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.160320 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.160495 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.161560 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.161658 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":133,"threads_started":1}
I20260925 22:49:51.162127 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.154958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.0.201.131:37177)
0925 22:49:51.155551 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.155563 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.155583 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:49:51.155659 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.155666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.155874 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.156027 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:51.156399 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.157637 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.159414 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.160230 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.160336 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.160497 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.160765 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.161991 (+ 1226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":157,"threads_started":1}
I20260925 22:49:51.162981 20660 tablet_service.cc:1467] Processing CreateTablet for tablet c2606551d4d44eabb5a7dabd31ee0349 (DEFAULT_TABLE table=test-table [id=871ed0d433bf44cf89d67093fc03aed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:51.164160 20660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2606551d4d44eabb5a7dabd31ee0349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:51.164587 20791 tablet_service.cc:1467] Processing CreateTablet for tablet c2606551d4d44eabb5a7dabd31ee0349 (DEFAULT_TABLE table=test-table [id=871ed0d433bf44cf89d67093fc03aed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:51.164868 20922 tablet_service.cc:1467] Processing CreateTablet for tablet c2606551d4d44eabb5a7dabd31ee0349 (DEFAULT_TABLE table=test-table [id=871ed0d433bf44cf89d67093fc03aed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:51.165477 20791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2606551d4d44eabb5a7dabd31ee0349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:51.165804 20922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2606551d4d44eabb5a7dabd31ee0349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:51.171255 21005 tablet_bootstrap.cc:492] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: Bootstrap starting.
I20260925 22:49:51.172221 21006 tablet_bootstrap.cc:492] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783: Bootstrap starting.
I20260925 22:49:51.173182 21007 tablet_bootstrap.cc:492] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: Bootstrap starting.
I20260925 22:49:51.174489 21006 tablet_bootstrap.cc:654] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:51.174489 21005 tablet_bootstrap.cc:654] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:51.174897 21007 tablet_bootstrap.cc:654] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:51.175371 21005 log.cc:824] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:51.175362 21006 log.cc:824] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:51.175598 21007 log.cc:824] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:51.177418 21005 tablet_bootstrap.cc:492] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: No bootstrap required, opened a new log
I20260925 22:49:51.177418 21007 tablet_bootstrap.cc:492] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: No bootstrap required, opened a new log
I20260925 22:49:51.177419 21006 tablet_bootstrap.cc:492] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783: No bootstrap required, opened a new log
I20260925 22:49:51.177609 21007 ts_tablet_manager.cc:1397] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:49:51.177609 21005 ts_tablet_manager.cc:1397] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260925 22:49:51.177620 21006 ts_tablet_manager.cc:1397] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:49:51.180720 21006 raft_consensus.cc:348] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63a504a703da430ebee5e9862cef8783" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39065 } } peers { permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 } } peers { permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 } }
I20260925 22:49:51.180964 21006 raft_consensus.cc:374] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:51.181018 21006 raft_consensus.cc:729] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a504a703da430ebee5e9862cef8783, State: Initialized, Role: FOLLOWER
I20260925 22:49:51.181450 21006 consensus_queue.cc:260] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63a504a703da430ebee5e9862cef8783" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39065 } } peers { permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 } } peers { permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 } }
I20260925 22:49:51.181834 21007 raft_consensus.cc:348] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63a504a703da430ebee5e9862cef8783" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39065 } } peers { permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 } } peers { permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 } }
I20260925 22:49:51.181834 21005 raft_consensus.cc:348] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63a504a703da430ebee5e9862cef8783" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39065 } } peers { permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 } } peers { permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 } }
I20260925 22:49:51.182046 20726 heartbeater.cc:503] Master 127.0.201.190:34777 was elected leader, sending a full tablet report...
I20260925 22:49:51.182075 21005 raft_consensus.cc:374] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:51.182075 21007 raft_consensus.cc:374] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:51.182132 21005 raft_consensus.cc:729] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6890e786a8b34582bf2220ba5afc0b0f, State: Initialized, Role: FOLLOWER
I20260925 22:49:51.182130 21007 raft_consensus.cc:729] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907deffd30eb4b8f80e1141836ea4bb8, State: Initialized, Role: FOLLOWER
I20260925 22:49:51.182406 21006 ts_tablet_manager.cc:1428] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260925 22:49:51.182530 21007 consensus_queue.cc:260] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63a504a703da430ebee5e9862cef8783" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39065 } } peers { permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 } } peers { permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 } }
I20260925 22:49:51.182530 21005 consensus_queue.cc:260] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63a504a703da430ebee5e9862cef8783" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39065 } } peers { permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 } } peers { permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 } }
I20260925 22:49:51.183387 20988 heartbeater.cc:503] Master 127.0.201.190:34777 was elected leader, sending a full tablet report...
I20260925 22:49:51.183419 21005 ts_tablet_manager.cc:1428] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260925 22:49:51.183835 20857 heartbeater.cc:503] Master 127.0.201.190:34777 was elected leader, sending a full tablet report...
I20260925 22:49:51.184330 21007 ts_tablet_manager.cc:1428] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:49:51.201920 21011 raft_consensus.cc:482] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:51.202116 21011 raft_consensus.cc:504] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63a504a703da430ebee5e9862cef8783" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39065 } } peers { permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 } } peers { permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 } }
I20260925 22:49:51.203198 21011 leader_election.cc:288] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6890e786a8b34582bf2220ba5afc0b0f (127.0.201.130:39531), 907deffd30eb4b8f80e1141836ea4bb8 (127.0.201.131:37177)
I20260925 22:49:51.207024 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.203368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.130:39531 (local address 127.0.201.129:41727)
0925 22:49:51.203565 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.203581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.203665 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.203977 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.203979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.203995 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:51.204158 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.204165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.205011 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.205015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.205900 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.205909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.206619 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.206623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.206723 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.206759 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.206831 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.206907 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1}
I20260925 22:49:51.207075 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.203496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:41727 (local address 127.0.201.130:39531)
0925 22:49:51.203645 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.203649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.203666 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:49:51.203808 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.203812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.203874 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.203957 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:51.204276 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.204902 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.206024 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.206503 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.206591 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.206627 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.206874 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.206942 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:49:51.207317 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.203616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.201.129:39275 (local address 127.0.201.131:37177)
0925 22:49:51.203752 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.203758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.204059 (+ 301us) server_negotiation.cc:407] Connection header received
0925 22:49:51.204183 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.204188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.204248 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.204325 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:51.204602 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.205174 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.206262 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.206715 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.206848 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.206888 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.207106 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.207183 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:49:51.207367 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.203565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.131:37177 (local address 127.0.201.129:39275)
0925 22:49:51.203957 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.203970 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.204073 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.204351 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.204354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.204382 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:51.204497 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.204502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.205286 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.205288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.206139 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.206149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.206873 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.206877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.206969 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.206990 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.207066 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.207136 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":26,"thread_start_us":144,"threads_started":1}
I20260925 22:49:51.207808 20811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2606551d4d44eabb5a7dabd31ee0349" candidate_uuid: "63a504a703da430ebee5e9862cef8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6890e786a8b34582bf2220ba5afc0b0f" is_pre_election: true
I20260925 22:49:51.207954 20942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2606551d4d44eabb5a7dabd31ee0349" candidate_uuid: "63a504a703da430ebee5e9862cef8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907deffd30eb4b8f80e1141836ea4bb8" is_pre_election: true
I20260925 22:49:51.208134 20811 raft_consensus.cc:2454] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63a504a703da430ebee5e9862cef8783 in term 0.
I20260925 22:49:51.208288 20942 raft_consensus.cc:2454] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63a504a703da430ebee5e9862cef8783 in term 0.
I20260925 22:49:51.208634 20615 leader_election.cc:302] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a504a703da430ebee5e9862cef8783, 6890e786a8b34582bf2220ba5afc0b0f; no voters:
W20260925 22:49:51.208829 20727 tablet.cc:2369] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:51.209131 21011 raft_consensus.cc:2781] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:51.209223 21011 raft_consensus.cc:482] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:51.209280 21011 raft_consensus.cc:3037] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:51.210253 21011 raft_consensus.cc:504] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63a504a703da430ebee5e9862cef8783" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39065 } } peers { permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 } } peers { permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 } }
I20260925 22:49:51.210600 21011 leader_election.cc:288] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [CANDIDATE]: Term 1 election: Requested vote from peers 6890e786a8b34582bf2220ba5afc0b0f (127.0.201.130:39531), 907deffd30eb4b8f80e1141836ea4bb8 (127.0.201.131:37177)
I20260925 22:49:51.210973 20811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2606551d4d44eabb5a7dabd31ee0349" candidate_uuid: "63a504a703da430ebee5e9862cef8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6890e786a8b34582bf2220ba5afc0b0f"
I20260925 22:49:51.210968 20942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2606551d4d44eabb5a7dabd31ee0349" candidate_uuid: "63a504a703da430ebee5e9862cef8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907deffd30eb4b8f80e1141836ea4bb8"
I20260925 22:49:51.211086 20811 raft_consensus.cc:3037] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:51.211086 20942 raft_consensus.cc:3037] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:51.212344 20811 raft_consensus.cc:2454] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63a504a703da430ebee5e9862cef8783 in term 1.
I20260925 22:49:51.212344 20942 raft_consensus.cc:2454] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63a504a703da430ebee5e9862cef8783 in term 1.
I20260925 22:49:51.212673 20614 leader_election.cc:302] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 63a504a703da430ebee5e9862cef8783, 6890e786a8b34582bf2220ba5afc0b0f, 907deffd30eb4b8f80e1141836ea4bb8; no voters:
I20260925 22:49:51.212860 21011 raft_consensus.cc:2781] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:51.213016 21011 raft_consensus.cc:686] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 1 LEADER]: Becoming Leader. State: Replica: 63a504a703da430ebee5e9862cef8783, State: Running, Role: LEADER
I20260925 22:49:51.213325 21011 consensus_queue.cc:237] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63a504a703da430ebee5e9862cef8783" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39065 } } peers { permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 } } peers { permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 } }
I20260925 22:49:51.216305 20538 catalog_manager.cc:5495] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a504a703da430ebee5e9862cef8783 (127.0.201.129). New cstate: current_term: 1 leader_uuid: "63a504a703da430ebee5e9862cef8783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63a504a703da430ebee5e9862cef8783" member_type: VOTER last_known_addr { host: "127.0.201.129" port: 39065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 } health_report { overall_health: UNKNOWN } } }
I20260925 22:49:51.229070 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.225738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.0.201.129:39065)
0925 22:49:51.225838 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.225847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.225878 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:49:51.226025 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.226028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.226079 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.226162 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:51.226368 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.226857 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.227630 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.227911 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.227990 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.228226 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:51.228636 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:51.228737 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.228809 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.228868 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 22:49:51.234418 20811 raft_consensus.cc:1261] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f [term 1 FOLLOWER]: Refusing update from remote peer 63a504a703da430ebee5e9862cef8783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:51.234421 20942 raft_consensus.cc:1261] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8 [term 1 FOLLOWER]: Refusing update from remote peer 63a504a703da430ebee5e9862cef8783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:51.235085 21011 consensus_queue.cc:1035] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6890e786a8b34582bf2220ba5afc0b0f" member_type: VOTER last_known_addr { host: "127.0.201.130" port: 39531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:51.235284 21015 consensus_queue.cc:1035] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "907deffd30eb4b8f80e1141836ea4bb8" member_type: VOTER last_known_addr { host: "127.0.201.131" port: 37177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:51.250232 20538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:51.250401 20538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:51.252709 20538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9555102add054b0ca1ad7c93ee9669b5: Sending DeleteTablet for 3 replicas of tablet c2606551d4d44eabb5a7dabd31ee0349
I20260925 22:49:51.253427 20791 tablet_service.cc:1514] Processing DeleteTablet for tablet c2606551d4d44eabb5a7dabd31ee0349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:51 UTC) from {username='slave'} at 127.0.0.1:50222
I20260925 22:49:51.253424 20660 tablet_service.cc:1514] Processing DeleteTablet for tablet c2606551d4d44eabb5a7dabd31ee0349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:51 UTC) from {username='slave'} at 127.0.0.1:46620
I20260925 22:49:51.253504 20922 tablet_service.cc:1514] Processing DeleteTablet for tablet c2606551d4d44eabb5a7dabd31ee0349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:51 UTC) from {username='slave'} at 127.0.0.1:57836
I20260925 22:49:51.253916 21029 tablet_replica.cc:330] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: stopping tablet replica
I20260925 22:49:51.254107 21028 tablet_replica.cc:330] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783: stopping tablet replica
I20260925 22:49:51.254410 21028 raft_consensus.cc:2229] T c2606551d4d44eabb5a7dabd31ee0349 P 63a504a703da430ebee5e9862cef8783 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:51.254590 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 20598
I20260925 22:49:51.255442 21029 raft_consensus.cc:2229] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:51.255793 21029 raft_consensus.cc:2258] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:51.256750 21030 tablet_replica.cc:330] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: stopping tablet replica
I20260925 22:49:51.257023 21030 raft_consensus.cc:2229] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:51.257282 21030 raft_consensus.cc:2258] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:51.257270 21029 ts_tablet_manager.cc:1905] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:51.258370 21030 ts_tablet_manager.cc:1905] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:51.260499 21030 ts_tablet_manager.cc:1918] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:51.260609 21030 log.cc:1196] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-2/wal/wals/c2606551d4d44eabb5a7dabd31ee0349
I20260925 22:49:51.260684 21029 ts_tablet_manager.cc:1918] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:51.260780 21029 log.cc:1196] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: Deleting WAL directory at /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376341615954-806-0/minicluster-data/ts-1/wal/wals/c2606551d4d44eabb5a7dabd31ee0349
I20260925 22:49:51.260954 21030 ts_tablet_manager.cc:1939] T c2606551d4d44eabb5a7dabd31ee0349 P 907deffd30eb4b8f80e1141836ea4bb8: Deleting consensus metadata
I20260925 22:49:51.261085 21029 ts_tablet_manager.cc:1939] T c2606551d4d44eabb5a7dabd31ee0349 P 6890e786a8b34582bf2220ba5afc0b0f: Deleting consensus metadata
I20260925 22:49:51.262060 20523 catalog_manager.cc:4841] TS 907deffd30eb4b8f80e1141836ea4bb8 (127.0.201.131:37177): tablet c2606551d4d44eabb5a7dabd31ee0349 (table test-table [id=871ed0d433bf44cf89d67093fc03aed7]) successfully deleted
I20260925 22:49:51.262178 20524 catalog_manager.cc:4841] TS 6890e786a8b34582bf2220ba5afc0b0f (127.0.201.130:39531): tablet c2606551d4d44eabb5a7dabd31ee0349 (table test-table [id=871ed0d433bf44cf89d67093fc03aed7]) successfully deleted
W20260925 22:49:51.262462 20525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.201.129:39065 (error 108)
I20260925 22:49:51.262741 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 20729
I20260925 22:49:51.264968 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.264675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.129:39065 (local address 127.0.0.1:46632)
0925 22:49:51.264828 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.264903 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.129:39065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:49:51.265113 20525 catalog_manager.cc:4584] TS 63a504a703da430ebee5e9862cef8783 (127.0.201.129:39065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2606551d4d44eabb5a7dabd31ee0349: Network error: Client connection negotiation failed: client connection to 127.0.201.129:39065: connect: Connection refused (error 111)
I20260925 22:49:51.268110 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 20860
I20260925 22:49:51.273711 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 20507
2026-09-25T22:49:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1010 ms)
[----------] 6 tests from EncryptionPolicyTest (10435 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:51 dist-test-slave-vdtf krb5kdc[21034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:51 dist-test-slave-vdtf krb5kdc[21034](info): set up 2 sockets
Sep 25 22:49:51 dist-test-slave-vdtf krb5kdc[21034](info): commencing operation
krb5kdc: starting...
W20260925 22:49:52.307861 806 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:49:52 dist-test-slave-vdtf krb5kdc[21034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:49:52.427139 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:52.534266 21047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:52.534562 21047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:52.534644 21047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:52.538651 21047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:52.538736 21047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:52.538774 21047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:52.538833 21047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:52.538873 21047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:52.544094 21047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:52.545094 21047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:52.546095 21047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:52.554184 21055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:52.554172 21053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:52.554173 21052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:52.554512 21047 server_base.cc:1034] running on GCE node
I20260925 22:49:52.555106 21047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:52.556227 21047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:52.557411 21047 hybrid_clock.cc:648] HybridClock initialized: now 1790376592557396 us; error 26 us; skew 500 ppm
Sep 25 22:49:52 dist-test-slave-vdtf krb5kdc[21034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:52.560065 21047 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:49:52.560899 21047 webserver.cc:460] Webserver started at http://127.0.0.1:46801/ using document root <none> and password file <none>
I20260925 22:49:52.561400 21047 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:52.561457 21047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:52.561618 21047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:52.563547 21047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "4dd606b0cd6f4a2b9b0997001636ce4b"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-vdtf"
server_key: "9d35b336c3e377144d2f1cb7bb5c03d8"
server_key_iv: "d30048953ca6ef61219dc2912205f09a"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.564133 21047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "4dd606b0cd6f4a2b9b0997001636ce4b"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-vdtf"
server_key: "9d35b336c3e377144d2f1cb7bb5c03d8"
server_key_iv: "d30048953ca6ef61219dc2912205f09a"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.567778 21047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:52.570212 21062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:52.571419 21047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:52.571516 21047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "4dd606b0cd6f4a2b9b0997001636ce4b"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-vdtf"
server_key: "9d35b336c3e377144d2f1cb7bb5c03d8"
server_key_iv: "d30048953ca6ef61219dc2912205f09a"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.571600 21047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:52.600383 21047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:52.603267 21047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:52.603425 21047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:52.610647 21047 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40249
I20260925 22:49:52.610669 21114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40249 every 8 connection(s)
I20260925 22:49:52.611604 21047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:52.613135 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 21047
I20260925 22:49:52.613287 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:52.613615 806 external_mini_cluster.cc:1428] Setting key b71f991ce9c95d3e6705369d917629f2
I20260925 22:49:52.614877 21115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:52.621368 21115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b: Bootstrap starting.
Sep 25 22:49:52 dist-test-slave-vdtf krb5kdc[21034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:49:52.623823 21115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:52.624526 21115 log.cc:824] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:52.626410 21115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b: No bootstrap required, opened a new log
I20260925 22:49:52.628258 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.615004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46271 (local address 127.0.0.1:40249)
0925 22:49:52.615403 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.615425 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.615460 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:49:52.616160 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.616177 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.616519 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.616798 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.617731 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:52.618697 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:52.619863 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:52.620325 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:52.620440 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:52.623212 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.623216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.623229 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:52.623255 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.625322 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:52.625837 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:52.625843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:52.625849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:52.625966 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:52.626304 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:52.626307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:52.626309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:52.626753 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.626976 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.627289 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.627506 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":142,"threads_started":1}
I20260925 22:49:52.630316 21115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd606b0cd6f4a2b9b0997001636ce4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } }
I20260925 22:49:52.630576 21115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:52.630645 21115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd606b0cd6f4a2b9b0997001636ce4b, State: Initialized, Role: FOLLOWER
I20260925 22:49:52.631052 21115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd606b0cd6f4a2b9b0997001636ce4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } }
I20260925 22:49:52.631253 21115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:52.631323 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:52.631386 21115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:52.631512 21115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:52.632440 21115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd606b0cd6f4a2b9b0997001636ce4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } }
I20260925 22:49:52.632783 21115 leader_election.cc:302] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dd606b0cd6f4a2b9b0997001636ce4b; no voters:
I20260925 22:49:52.633080 21115 leader_election.cc:288] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:52.633450 21125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:52.633800 21125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [term 1 LEADER]: Becoming Leader. State: Replica: 4dd606b0cd6f4a2b9b0997001636ce4b, State: Running, Role: LEADER
I20260925 22:49:52.634124 21125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd606b0cd6f4a2b9b0997001636ce4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } }
I20260925 22:49:52.634435 21115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:52.635233 21125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dd606b0cd6f4a2b9b0997001636ce4b. Latest consensus state: current_term: 1 leader_uuid: "4dd606b0cd6f4a2b9b0997001636ce4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd606b0cd6f4a2b9b0997001636ce4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } } }
I20260925 22:49:52.635341 21125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:52.635664 21134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:52.636358 21124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dd606b0cd6f4a2b9b0997001636ce4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd606b0cd6f4a2b9b0997001636ce4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } } }
I20260925 22:49:52.636531 21124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:52.638943 21134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:52.642550 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.633445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37449 (local address 127.0.0.1:40249)
0925 22:49:52.633547 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.633551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.633723 (+ 172us) server_negotiation.cc:407] Connection header received
0925 22:49:52.633895 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.633898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.633941 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.633989 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.634886 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:52.635713 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:52.636885 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:52.637209 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:52.637312 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:52.638797 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.638801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.638804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:52.638829 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.640635 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:52.641195 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:52.641198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:52.641200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:52.641292 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:52.641704 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:52.641707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:52.641709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:52.641891 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.642042 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.642309 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.642399 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 22:49:52.645421 21134 catalog_manager.cc:1347] Generated new cluster ID: cc88dd3c877445fa9ac36f1db13c23cd
I20260925 22:49:52.645478 21134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:52.666244 21134 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:52.667513 21134 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:52.675046 21134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dd606b0cd6f4a2b9b0997001636ce4b: Generated new TSK 0
I20260925 22:49:52.675866 21134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:52.683533 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 21047
W20260925 22:49:52.689316 21032 connection.cc:535] client connection to 127.0.0.1:40249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40249): Connection reset by peer (error 104)
2026-09-25T22:49:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:52 dist-test-slave-vdtf krb5kdc[21146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:52 dist-test-slave-vdtf krb5kdc[21146](info): set up 2 sockets
Sep 25 22:49:52 dist-test-slave-vdtf krb5kdc[21146](info): commencing operation
krb5kdc: starting...
W20260925 22:49:53.719827 806 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:49:53 dist-test-slave-vdtf krb5kdc[21146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:49:53.839566 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:53.949664 21159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:53.949997 21159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:53.950079 21159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:53.954169 21159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:53.954252 21159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:53.954293 21159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:53.954348 21159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:53.954385 21159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:53.959666 21159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42735
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:53.960788 21159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:53.961862 21159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:53.970274 21164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:53.970433 21159 server_base.cc:1034] running on GCE node
W20260925 22:49:53.970335 21165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:53.970609 21167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:53.971240 21159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:53.972394 21159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:53.973580 21159 hybrid_clock.cc:648] HybridClock initialized: now 1790376593973528 us; error 66 us; skew 500 ppm
Sep 25 22:49:53 dist-test-slave-vdtf krb5kdc[21146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:53.976399 21159 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:49:53.977298 21159 webserver.cc:460] Webserver started at http://127.0.0.1:44723/ using document root <none> and password file <none>
I20260925 22:49:53.977875 21159 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:53.977964 21159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:53.978197 21159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:53.980331 21159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "1ba006132a80441e8869be04f2e3cc72"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-vdtf"
server_key: "5dc8105a834a74f6a4a87d1786b097b4"
server_key_iv: "b541770d0b50ce0395051a4f715684e7"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.980886 21159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "1ba006132a80441e8869be04f2e3cc72"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-vdtf"
server_key: "5dc8105a834a74f6a4a87d1786b097b4"
server_key_iv: "b541770d0b50ce0395051a4f715684e7"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.984653 21159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:49:53.987007 21174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:53.988205 21159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:53.988343 21159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "1ba006132a80441e8869be04f2e3cc72"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-vdtf"
server_key: "5dc8105a834a74f6a4a87d1786b097b4"
server_key_iv: "b541770d0b50ce0395051a4f715684e7"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.988452 21159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:54.005854 21159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:54.009012 21159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:54.009222 21159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:54.015870 21159 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42735
I20260925 22:49:54.015897 21226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42735 every 8 connection(s)
I20260925 22:49:54.016866 21159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:54.019649 21227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:54.025058 21227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72: Bootstrap starting.
I20260925 22:49:54.026430 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 21159
I20260925 22:49:54.026582 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:54.026887 806 external_mini_cluster.cc:1428] Setting key 77e23a70a9605edc8e82573dac9abd9e
I20260925 22:49:54.027763 21227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:54.028738 21227 log.cc:824] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:54.030601 21227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72: No bootstrap required, opened a new log
I20260925 22:49:54.033951 21227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba006132a80441e8869be04f2e3cc72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42735 } }
I20260925 22:49:54.034189 21227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:54.034281 21227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba006132a80441e8869be04f2e3cc72, State: Initialized, Role: FOLLOWER
I20260925 22:49:54.034796 21227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba006132a80441e8869be04f2e3cc72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42735 } }
I20260925 22:49:54.034946 21227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:54.035017 21227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:54.035118 21227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:54.036036 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:54.028191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38311 (local address 127.0.0.1:42735)
0925 22:49:54.028641 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:54.028656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:54.028705 (+ 49us) server_negotiation.cc:407] Connection header received
0925 22:49:54.029616 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:54.029634 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:54.029970 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:54.030319 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:54.030736 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:54.031761 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:54.032657 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:54.033143 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:54.033315 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:54.034216 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:54.034223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:54.034239 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:54.034701 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:54.034753 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:54.034775 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:54.034922 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:54.035072 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 22:49:54.035211 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":138,"threads_started":1}
I20260925 22:49:54.036118 21227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba006132a80441e8869be04f2e3cc72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42735 } }
I20260925 22:49:54.036518 21227 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba006132a80441e8869be04f2e3cc72; no voters:
I20260925 22:49:54.036832 21227 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:54.036947 21232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:54.037165 21232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [term 1 LEADER]: Becoming Leader. State: Replica: 1ba006132a80441e8869be04f2e3cc72, State: Running, Role: LEADER
I20260925 22:49:54.037511 21232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba006132a80441e8869be04f2e3cc72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42735 } }
I20260925 22:49:54.038110 21227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:54.039386 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:54.039340 21233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba006132a80441e8869be04f2e3cc72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba006132a80441e8869be04f2e3cc72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42735 } } }
I20260925 22:49:54.039386 21234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba006132a80441e8869be04f2e3cc72. Latest consensus state: current_term: 1 leader_uuid: "1ba006132a80441e8869be04f2e3cc72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba006132a80441e8869be04f2e3cc72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42735 } } }
I20260925 22:49:54.039456 21233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:54.039481 21234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:54.040149 21241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:54.043159 21241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:54.046367 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:54.041835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38711 (local address 127.0.0.1:42735)
0925 22:49:54.041949 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:54.041953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:54.041973 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:49:54.042108 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:54.042111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:54.042154 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:54.042234 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:54.042572 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:54.043332 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:54.044473 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:54.044830 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:54.044937 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:54.045876 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:54.045880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:54.045883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:54.045896 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:54.045924 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:54.045937 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:54.046021 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:54.046196 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:49:54.046230 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:49:54.048682 21241 catalog_manager.cc:1347] Generated new cluster ID: 34405bfdf1844eb6b14a6725123ab994
I20260925 22:49:54.048755 21241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:54.084574 21241 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:54.085646 21241 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:54.090837 21241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ba006132a80441e8869be04f2e3cc72: Generated new TSK 0
I20260925 22:49:54.091594 21241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:54.100679 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 21159
W20260925 22:49:54.106026 21144 connection.cc:535] client connection to 127.0.0.1:42735 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42735): Connection reset by peer (error 104)
2026-09-25T22:49:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:54 dist-test-slave-vdtf krb5kdc[21258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:54 dist-test-slave-vdtf krb5kdc[21258](info): set up 2 sockets
Sep 25 22:49:54 dist-test-slave-vdtf krb5kdc[21258](info): commencing operation
krb5kdc: starting...
W20260925 22:49:55.127921 806 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:49:55 dist-test-slave-vdtf krb5kdc[21258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:49:55.247591 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:55.358879 21271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:55.359175 21271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:55.359246 21271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:55.363188 21271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:55.363286 21271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:55.363324 21271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:55.363363 21271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:55.363397 21271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:55.368507 21271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:55.369571 21271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:55.370594 21271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:55.379015 21279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:55.379031 21277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:55.379015 21276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:55.379395 21271 server_base.cc:1034] running on GCE node
I20260925 22:49:55.380015 21271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:55.381083 21271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:55.382269 21271 hybrid_clock.cc:648] HybridClock initialized: now 1790376595382264 us; error 31 us; skew 500 ppm
Sep 25 22:49:55 dist-test-slave-vdtf krb5kdc[21258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:55.385149 21271 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:49:55.386009 21271 webserver.cc:460] Webserver started at http://127.0.0.1:36825/ using document root <none> and password file <none>
I20260925 22:49:55.386564 21271 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:55.386653 21271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:55.386889 21271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:55.388996 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "210e5124f20445f4a561477b2ebbb324"
format_stamp: "Formatted at 2026-09-25 22:49:55 on dist-test-slave-vdtf"
server_key: "3206749fa87971f0b933f1406bed39c7"
server_key_iv: "c8f08e6fb04b2dfd535c53ec39994e55"
server_key_version: "encryptionkey@0"
I20260925 22:49:55.389554 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "210e5124f20445f4a561477b2ebbb324"
format_stamp: "Formatted at 2026-09-25 22:49:55 on dist-test-slave-vdtf"
server_key: "3206749fa87971f0b933f1406bed39c7"
server_key_iv: "c8f08e6fb04b2dfd535c53ec39994e55"
server_key_version: "encryptionkey@0"
I20260925 22:49:55.393224 21271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:55.395560 21286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:55.396824 21271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:55.396984 21271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "210e5124f20445f4a561477b2ebbb324"
format_stamp: "Formatted at 2026-09-25 22:49:55 on dist-test-slave-vdtf"
server_key: "3206749fa87971f0b933f1406bed39c7"
server_key_iv: "c8f08e6fb04b2dfd535c53ec39994e55"
server_key_version: "encryptionkey@0"
I20260925 22:49:55.397095 21271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:55.401577 21271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:55.404838 21271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:55.405037 21271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:55.411586 21271 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36539
I20260925 22:49:55.411665 21338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36539 every 8 connection(s)
I20260925 22:49:55.412523 21271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:55.413187 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 21271
I20260925 22:49:55.413343 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:55.413645 806 external_mini_cluster.cc:1428] Setting key 182c5eb582535bda9319db6a41c713ed
I20260925 22:49:55.415817 21339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:55.419117 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:55.415001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43751 (local address 127.0.0.1:36539)
0925 22:49:55.415439 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:55.415459 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:55.415502 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:49:55.416359 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:55.416381 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:55.416730 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:55.417086 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:55.417751 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:55.417756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:55.417771 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:55.417782 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:55.417840 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:55.417868 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:55.418007 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:55.418074 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 22:49:55.418221 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":106,"threads_started":1}
I20260925 22:49:55.422008 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:55.422634 21339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324: Bootstrap starting.
I20260925 22:49:55.425191 21339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:55.425772 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:55.424111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35565 (local address 127.0.0.1:36539)
0925 22:49:55.424238 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:55.424243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:55.424439 (+ 196us) server_negotiation.cc:407] Connection header received
0925 22:49:55.424568 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:55.424570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:55.424598 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:55.424655 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:55.425350 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:55.425353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:55.425356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:55.425369 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:55.425397 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:55.425407 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:55.425474 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:55.425593 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 22:49:55.425632 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:55.426098 21339 log.cc:824] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:55.428069 21339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324: No bootstrap required, opened a new log
I20260925 22:49:55.431399 21339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e5124f20445f4a561477b2ebbb324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } }
I20260925 22:49:55.431612 21339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:55.431707 21339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210e5124f20445f4a561477b2ebbb324, State: Initialized, Role: FOLLOWER
I20260925 22:49:55.432193 21339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e5124f20445f4a561477b2ebbb324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } }
I20260925 22:49:55.432349 21339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:55.432415 21339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:55.432512 21339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:55.433367 21339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e5124f20445f4a561477b2ebbb324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } }
I20260925 22:49:55.433700 21339 leader_election.cc:302] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 210e5124f20445f4a561477b2ebbb324; no voters:
I20260925 22:49:55.434000 21339 leader_election.cc:288] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:55.434109 21349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:55.434374 21349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [term 1 LEADER]: Becoming Leader. State: Replica: 210e5124f20445f4a561477b2ebbb324, State: Running, Role: LEADER
I20260925 22:49:55.434693 21349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e5124f20445f4a561477b2ebbb324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } }
I20260925 22:49:55.435226 21339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:55.436313 21351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 210e5124f20445f4a561477b2ebbb324. Latest consensus state: current_term: 1 leader_uuid: "210e5124f20445f4a561477b2ebbb324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e5124f20445f4a561477b2ebbb324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } } }
I20260925 22:49:55.436444 21351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:55.436781 21358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:55.436776 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "210e5124f20445f4a561477b2ebbb324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e5124f20445f4a561477b2ebbb324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } } }
I20260925 22:49:55.436937 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:55.439460 21358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:55.444756 21358 catalog_manager.cc:1347] Generated new cluster ID: cee3969ee6614fac90be94a568d20eef
I20260925 22:49:55.444823 21358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:55.463034 21358 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:55.463356 21358 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:55.479346 21358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 210e5124f20445f4a561477b2ebbb324: Generated new TSK 0
I20260925 22:49:55.480257 21358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:55.495833 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 21271
2026-09-25T22:49:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:55 dist-test-slave-vdtf krb5kdc[21370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:55 dist-test-slave-vdtf krb5kdc[21370](info): set up 2 sockets
Sep 25 22:49:55 dist-test-slave-vdtf krb5kdc[21370](info): commencing operation
krb5kdc: starting...
W20260925 22:49:56.535907 806 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:49:56 dist-test-slave-vdtf krb5kdc[21370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:49:56.655351 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:56.763574 21383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:56.763913 21383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:56.763998 21383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:56.767956 21383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:56.768050 21383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:56.768088 21383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:56.768141 21383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:56.768178 21383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:56.773394 21383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:56.774425 21383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:56.775439 21383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:56.783947 21389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:56.783905 21391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:56.783962 21383 server_base.cc:1034] running on GCE node
W20260925 22:49:56.783905 21388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:56.784667 21383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:56.785746 21383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:56.786932 21383 hybrid_clock.cc:648] HybridClock initialized: now 1790376596786907 us; error 43 us; skew 500 ppm
Sep 25 22:49:56 dist-test-slave-vdtf krb5kdc[21370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.799351 21383 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:49:56.800477 21383 webserver.cc:460] Webserver started at http://127.0.0.1:40269/ using document root <none> and password file <none>
I20260925 22:49:56.801071 21383 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:56.801167 21383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:56.801388 21383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:56.803473 21383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "5c4a907a7096477e94ca9dea7f4841a0"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-vdtf"
server_key: "180985cffe8ced2133f2c6f84ebd8c17"
server_key_iv: "bc23dd9bd3bec1962d0c771d35ec7601"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.804105 21383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "5c4a907a7096477e94ca9dea7f4841a0"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-vdtf"
server_key: "180985cffe8ced2133f2c6f84ebd8c17"
server_key_iv: "bc23dd9bd3bec1962d0c771d35ec7601"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.807890 21383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:49:56.810294 21398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:56.811564 21383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:56.811748 21383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "5c4a907a7096477e94ca9dea7f4841a0"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-vdtf"
server_key: "180985cffe8ced2133f2c6f84ebd8c17"
server_key_iv: "bc23dd9bd3bec1962d0c771d35ec7601"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.811868 21383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:56.835708 21383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:56.838583 21383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:56.838800 21383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:56.845918 21383 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45453
I20260925 22:49:56.845939 21450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45453 every 8 connection(s)
I20260925 22:49:56.846874 21383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:56.849674 21451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:56.851742 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 21383
I20260925 22:49:56.851888 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:56.852181 806 external_mini_cluster.cc:1428] Setting key 3223afe5d4a6c70b19d8ecd26497a63d
I20260925 22:49:56.856418 21451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0: Bootstrap starting.
I20260925 22:49:56.858840 21451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:56.859750 21451 log.cc:824] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:49:56 dist-test-slave-vdtf krb5kdc[21370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:49:56.861735 21451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0: No bootstrap required, opened a new log
I20260925 22:49:56.866369 21451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4a907a7096477e94ca9dea7f4841a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45453 } }
I20260925 22:49:56.866612 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.853538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34939 (local address 127.0.0.1:45453)
0925 22:49:56.853927 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.853948 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.853985 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:49:56.854612 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.854628 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.854961 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.855232 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.856164 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.857192 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.858374 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.858824 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.858987 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.861657 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:56.861663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:56.861678 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:56.861706 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:56.863611 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.864077 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.864085 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.864093 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.864255 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.864512 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.864515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.864517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.865004 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:56.865206 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:56.865381 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:49:56.865677 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260925 22:49:56.866631 21451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:56.866676 21451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4a907a7096477e94ca9dea7f4841a0, State: Initialized, Role: FOLLOWER
I20260925 22:49:56.867096 21451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4a907a7096477e94ca9dea7f4841a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45453 } }
I20260925 22:49:56.867218 21451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:56.867297 21451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:56.867449 21451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:56.868360 21451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4a907a7096477e94ca9dea7f4841a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45453 } }
I20260925 22:49:56.868710 21451 leader_election.cc:302] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c4a907a7096477e94ca9dea7f4841a0; no voters:
I20260925 22:49:56.869058 21451 leader_election.cc:288] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:56.869130 21456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:56.869328 21456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [term 1 LEADER]: Becoming Leader. State: Replica: 5c4a907a7096477e94ca9dea7f4841a0, State: Running, Role: LEADER
I20260925 22:49:56.869640 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:56.869658 21456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4a907a7096477e94ca9dea7f4841a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45453 } }
I20260925 22:49:56.870286 21451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:56.870877 21456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4a907a7096477e94ca9dea7f4841a0. Latest consensus state: current_term: 1 leader_uuid: "5c4a907a7096477e94ca9dea7f4841a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4a907a7096477e94ca9dea7f4841a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45453 } } }
I20260925 22:49:56.871013 21456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:56.871279 21457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c4a907a7096477e94ca9dea7f4841a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4a907a7096477e94ca9dea7f4841a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45453 } } }
I20260925 22:49:56.871387 21457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:56.872270 21467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:56.875617 21467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:56.882102 21467 catalog_manager.cc:1347] Generated new cluster ID: 1663d97855024e07b0a94508058d6bb5
I20260925 22:49:56.882225 21467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:56.882316 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.871938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50215 (local address 127.0.0.1:45453)
0925 22:49:56.872053 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.872059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.872656 (+ 597us) server_negotiation.cc:407] Connection header received
0925 22:49:56.872965 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.872968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.873017 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.873095 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.873954 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.874714 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.876524 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.876903 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.877016 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.878590 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:56.878594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:56.878597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:56.878621 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:56.880165 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.880668 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.880672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.880674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.880758 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.881065 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.881069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.881070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.881279 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:56.881399 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:56.882002 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 22:49:56.882133 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:49:56.922246 21467 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:56.923529 21467 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:56.943387 21467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c4a907a7096477e94ca9dea7f4841a0: Generated new TSK 0
I20260925 22:49:56.944205 21467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:56.956622 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 21383
2026-09-25T22:49:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:57 dist-test-slave-vdtf krb5kdc[21482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:57 dist-test-slave-vdtf krb5kdc[21482](info): set up 2 sockets
Sep 25 22:49:57 dist-test-slave-vdtf krb5kdc[21482](info): commencing operation
krb5kdc: starting...
W20260925 22:49:58.000284 806 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:49:58 dist-test-slave-vdtf krb5kdc[21482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:49:58.120980 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:58.228933 21495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:58.229209 21495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:58.229302 21495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:58.233331 21495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:58.233405 21495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:58.233453 21495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:58.233485 21495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:58.233538 21495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:58.238734 21495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:58.239822 21495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:58.240805 21495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:58.249012 21501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:58.249025 21500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:58.249287 21495 server_base.cc:1034] running on GCE node
W20260925 22:49:58.249082 21503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:58.249959 21495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:58.250941 21495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:58.252099 21495 hybrid_clock.cc:648] HybridClock initialized: now 1790376598252071 us; error 41 us; skew 500 ppm
Sep 25 22:49:58 dist-test-slave-vdtf krb5kdc[21482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:58.254685 21495 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:49:58.255517 21495 webserver.cc:460] Webserver started at http://127.0.0.1:33451/ using document root <none> and password file <none>
I20260925 22:49:58.256068 21495 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:58.256129 21495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:58.256332 21495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:58.258685 21495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "68d0e224b90c4ffc9305fea92c344300"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-vdtf"
server_key: "27df75a7e14fb5e1c7041c9463788aca"
server_key_iv: "f1404ad49f2cfc63d601ee411bd51185"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.259407 21495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "68d0e224b90c4ffc9305fea92c344300"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-vdtf"
server_key: "27df75a7e14fb5e1c7041c9463788aca"
server_key_iv: "f1404ad49f2cfc63d601ee411bd51185"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.264219 21495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:49:58.266361 21510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.267511 21495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:58.267603 21495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "68d0e224b90c4ffc9305fea92c344300"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-vdtf"
server_key: "27df75a7e14fb5e1c7041c9463788aca"
server_key_iv: "f1404ad49f2cfc63d601ee411bd51185"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.267735 21495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:58.279019 21495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:58.281880 21495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:58.282020 21495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:58.289099 21495 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43811
I20260925 22:49:58.289122 21562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43811 every 8 connection(s)
I20260925 22:49:58.289979 21495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:58.292654 21563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:58.296981 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 21495
I20260925 22:49:58.297094 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:58.297376 806 external_mini_cluster.cc:1428] Setting key 0df55f8dcb659fcbed2e36be4952a0e0
I20260925 22:49:58.298226 21563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300: Bootstrap starting.
I20260925 22:49:58.300823 21563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:58.301697 21563 log.cc:824] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:58.303923 21563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300: No bootstrap required, opened a new log
I20260925 22:49:58.307289 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.298717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59365 (local address 127.0.0.1:43811)
0925 22:49:58.299125 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:58.299143 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:58.299185 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:49:58.300028 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:58.300051 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:58.300469 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:58.300847 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:58.301281 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.302604 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.303539 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.304248 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.304416 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.305227 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:58.305233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:58.305250 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:58.305750 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:58.305818 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:58.305848 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:58.305950 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:58.306059 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 22:49:58.306259 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":108,"threads_started":1}
I20260925 22:49:58.308561 21563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0e224b90c4ffc9305fea92c344300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43811 } }
I20260925 22:49:58.308768 21563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:58.308856 21563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d0e224b90c4ffc9305fea92c344300, State: Initialized, Role: FOLLOWER
I20260925 22:49:58.309345 21563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0e224b90c4ffc9305fea92c344300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43811 } }
I20260925 22:49:58.309526 21563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:58.309592 21563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:58.309690 21563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:58.310170 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:58.310655 21563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0e224b90c4ffc9305fea92c344300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43811 } }
I20260925 22:49:58.310984 21563 leader_election.cc:302] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68d0e224b90c4ffc9305fea92c344300; no voters:
I20260925 22:49:58.311334 21563 leader_election.cc:288] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:58.311367 21569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:58.311560 21569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [term 1 LEADER]: Becoming Leader. State: Replica: 68d0e224b90c4ffc9305fea92c344300, State: Running, Role: LEADER
I20260925 22:49:58.311872 21569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0e224b90c4ffc9305fea92c344300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43811 } }
I20260925 22:49:58.312543 21563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:58.312999 21569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68d0e224b90c4ffc9305fea92c344300. Latest consensus state: current_term: 1 leader_uuid: "68d0e224b90c4ffc9305fea92c344300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0e224b90c4ffc9305fea92c344300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43811 } } }
I20260925 22:49:58.313109 21569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:58.313316 21569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68d0e224b90c4ffc9305fea92c344300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0e224b90c4ffc9305fea92c344300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43811 } } }
I20260925 22:49:58.313417 21569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:58.313602 21578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:58.317481 21578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:58.318259 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.312218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37065 (local address 127.0.0.1:43811)
0925 22:49:58.312332 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:58.312336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:58.313906 (+ 1570us) server_negotiation.cc:407] Connection header received
0925 22:49:58.314082 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:58.314085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:58.314142 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:58.314199 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:58.314700 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.315340 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.316442 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.316762 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.316871 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.317734 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:58.317737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:58.317740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:58.317753 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:58.317780 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:58.317792 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:58.317837 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:58.318086 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 22:49:58.318143 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:49:58.323153 21578 catalog_manager.cc:1347] Generated new cluster ID: 1d411d47e5a64b92a024d3383b698e26
I20260925 22:49:58.323215 21578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:58.358284 21578 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:58.359575 21578 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:58.368054 21578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68d0e224b90c4ffc9305fea92c344300: Generated new TSK 0
I20260925 22:49:58.368896 21578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:58.381767 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 21495
2026-09-25T22:49:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:58 dist-test-slave-vdtf krb5kdc[21594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:58 dist-test-slave-vdtf krb5kdc[21594](info): set up 2 sockets
Sep 25 22:49:58 dist-test-slave-vdtf krb5kdc[21594](info): commencing operation
krb5kdc: starting...
W20260925 22:49:59.415975 806 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:49:59 dist-test-slave-vdtf krb5kdc[21594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:49:59.535585 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:59.646448 21607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:59.646703 21607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:59.646745 21607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:59.650808 21607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:59.650866 21607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:59.650910 21607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:59.650951 21607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:59.650990 21607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:59.656668 21607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:49:59.657688 21607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:59.658697 21607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:59.667074 21613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.667143 21612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.667163 21615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:59.667469 21607 server_base.cc:1034] running on GCE node
I20260925 22:49:59.668071 21607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:59.669148 21607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:59.670337 21607 hybrid_clock.cc:648] HybridClock initialized: now 1790376599670318 us; error 29 us; skew 500 ppm
Sep 25 22:49:59 dist-test-slave-vdtf krb5kdc[21594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:59.673071 21607 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:49:59.673919 21607 webserver.cc:460] Webserver started at http://127.0.0.1:46247/ using document root <none> and password file <none>
I20260925 22:49:59.674460 21607 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:59.674548 21607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:59.674772 21607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:59.676792 21607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "7f55f4bba6754299b369ecffdb7e3c6f"
format_stamp: "Formatted at 2026-09-25 22:49:59 on dist-test-slave-vdtf"
server_key: "eea27f0d803770a24887703a891ef6e7"
server_key_iv: "d38428e9b2fd6773370fbfff3e80fcab"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.677388 21607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "7f55f4bba6754299b369ecffdb7e3c6f"
format_stamp: "Formatted at 2026-09-25 22:49:59 on dist-test-slave-vdtf"
server_key: "eea27f0d803770a24887703a891ef6e7"
server_key_iv: "d38428e9b2fd6773370fbfff3e80fcab"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.680961 21607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:59.683247 21622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.684715 21607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:59.684866 21607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "7f55f4bba6754299b369ecffdb7e3c6f"
format_stamp: "Formatted at 2026-09-25 22:49:59 on dist-test-slave-vdtf"
server_key: "eea27f0d803770a24887703a891ef6e7"
server_key_iv: "d38428e9b2fd6773370fbfff3e80fcab"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.684980 21607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:59.689458 21607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:59.692227 21607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:59.692427 21607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:59.699198 21607 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40973
I20260925 22:49:59.699265 21674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40973 every 8 connection(s)
I20260925 22:49:59.700172 21607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:59.701574 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 21607
I20260925 22:49:59.701725 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:49:59.702020 806 external_mini_cluster.cc:1428] Setting key c4885527aa1d5a8862ad5a10a334dccd
I20260925 22:49:59.703250 21675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:59.707252 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.703404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37845 (local address 127.0.0.1:40973)
0925 22:49:59.703851 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.703866 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.703900 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:49:59.704519 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.704533 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.704818 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.705148 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.706131 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.706135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.706147 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:59.706158 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.706210 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:59.706228 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.706372 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.706443 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.706603 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":131,"threads_started":1}
I20260925 22:49:59.709136 21675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f: Bootstrap starting.
I20260925 22:49:59.709967 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:59.711469 21675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:59.712419 21675 log.cc:824] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:59.713832 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.711997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45499 (local address 127.0.0.1:40973)
0925 22:49:59.712115 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.712119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.712371 (+ 252us) server_negotiation.cc:407] Connection header received
0925 22:49:59.712497 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.712499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.712549 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.712632 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.713432 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.713435 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.713436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:59.713445 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.713468 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:59.713477 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.713555 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.713657 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.713685 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:49:59.714545 21675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f: No bootstrap required, opened a new log
I20260925 22:49:59.717922 21675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f55f4bba6754299b369ecffdb7e3c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260925 22:49:59.718152 21675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:59.718214 21675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f55f4bba6754299b369ecffdb7e3c6f, State: Initialized, Role: FOLLOWER
I20260925 22:49:59.718598 21675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f55f4bba6754299b369ecffdb7e3c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260925 22:49:59.718740 21675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:59.718825 21675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:59.718933 21675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:59.719818 21675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f55f4bba6754299b369ecffdb7e3c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260925 22:49:59.720141 21675 leader_election.cc:302] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f55f4bba6754299b369ecffdb7e3c6f; no voters:
I20260925 22:49:59.720455 21675 leader_election.cc:288] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:59.720522 21685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:59.720718 21685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [term 1 LEADER]: Becoming Leader. State: Replica: 7f55f4bba6754299b369ecffdb7e3c6f, State: Running, Role: LEADER
I20260925 22:49:59.720986 21685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f55f4bba6754299b369ecffdb7e3c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260925 22:49:59.721668 21675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:59.722599 21687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f55f4bba6754299b369ecffdb7e3c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f55f4bba6754299b369ecffdb7e3c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } } }
I20260925 22:49:59.722723 21687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:59.722990 21686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f55f4bba6754299b369ecffdb7e3c6f. Latest consensus state: current_term: 1 leader_uuid: "7f55f4bba6754299b369ecffdb7e3c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f55f4bba6754299b369ecffdb7e3c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } } }
I20260925 22:49:59.723089 21694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:59.723145 21686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:59.725703 21694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:59.730811 21694 catalog_manager.cc:1347] Generated new cluster ID: b393906530714b70be4bfebc8d001884
I20260925 22:49:59.730862 21694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:59.740211 21694 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:59.740520 21694 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:59.752523 21694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f55f4bba6754299b369ecffdb7e3c6f: Generated new TSK 0
I20260925 22:49:59.753193 21694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:59.761096 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 21607
2026-09-25T22:49:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:59 dist-test-slave-vdtf krb5kdc[21706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:59 dist-test-slave-vdtf krb5kdc[21706](info): set up 2 sockets
Sep 25 22:49:59 dist-test-slave-vdtf krb5kdc[21706](info): commencing operation
krb5kdc: starting...
W20260925 22:50:00.795980 806 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:00 dist-test-slave-vdtf krb5kdc[21706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:50:00.915416 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:01.034922 21719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:01.035209 21719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:01.035290 21719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:01.039356 21719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:01.039443 21719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:01.039484 21719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:01.039521 21719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:01.039557 21719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:01.044857 21719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:01.045869 21719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:01.046864 21719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:01.055265 21724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:01.055316 21727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:01.055353 21719 server_base.cc:1034] running on GCE node
W20260925 22:50:01.055317 21725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:01.056078 21719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:01.057086 21719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:01.058276 21719 hybrid_clock.cc:648] HybridClock initialized: now 1790376601058257 us; error 30 us; skew 500 ppm
Sep 25 22:50:01 dist-test-slave-vdtf krb5kdc[21706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:01.061087 21719 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:50:01.061908 21719 webserver.cc:460] Webserver started at http://127.0.0.1:36615/ using document root <none> and password file <none>
I20260925 22:50:01.062455 21719 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:01.062549 21719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:01.062777 21719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:01.064838 21719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "1481e5a4cae84c96923a5fc87d863916"
format_stamp: "Formatted at 2026-09-25 22:50:01 on dist-test-slave-vdtf"
server_key: "3a7004f3f15c8222d81c6448e500880c"
server_key_iv: "dd6aa8667c335f990a01a5590bc4c55e"
server_key_version: "encryptionkey@0"
I20260925 22:50:01.065390 21719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "1481e5a4cae84c96923a5fc87d863916"
format_stamp: "Formatted at 2026-09-25 22:50:01 on dist-test-slave-vdtf"
server_key: "3a7004f3f15c8222d81c6448e500880c"
server_key_iv: "dd6aa8667c335f990a01a5590bc4c55e"
server_key_version: "encryptionkey@0"
I20260925 22:50:01.069201 21719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:50:01.071607 21734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:01.072875 21719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:01.073031 21719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "1481e5a4cae84c96923a5fc87d863916"
format_stamp: "Formatted at 2026-09-25 22:50:01 on dist-test-slave-vdtf"
server_key: "3a7004f3f15c8222d81c6448e500880c"
server_key_iv: "dd6aa8667c335f990a01a5590bc4c55e"
server_key_version: "encryptionkey@0"
I20260925 22:50:01.073151 21719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:01.090319 21719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:01.093556 21719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:01.093773 21719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:01.100806 21719 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41329
I20260925 22:50:01.100873 21786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41329 every 8 connection(s)
I20260925 22:50:01.101848 21719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:01.104692 21787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:01.110257 21787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916: Bootstrap starting.
I20260925 22:50:01.111418 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 21719
I20260925 22:50:01.111595 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:01.111898 806 external_mini_cluster.cc:1428] Setting key 105a2ed9db76a808f2364e62cf2aa226
I20260925 22:50:01.112998 21787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:01.113742 21787 log.cc:824] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:01.116008 21787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916: No bootstrap required, opened a new log
I20260925 22:50:01.120371 21787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1481e5a4cae84c96923a5fc87d863916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41329 } }
I20260925 22:50:01.120707 21787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:01.120770 21787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1481e5a4cae84c96923a5fc87d863916, State: Initialized, Role: FOLLOWER
Sep 25 22:50:01 dist-test-slave-vdtf krb5kdc[21706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:50:01.121201 21787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1481e5a4cae84c96923a5fc87d863916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41329 } }
I20260925 22:50:01.121371 21787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:01.121461 21787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:01.121558 21787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:01.122643 21787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1481e5a4cae84c96923a5fc87d863916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41329 } }
I20260925 22:50:01.123030 21787 leader_election.cc:302] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1481e5a4cae84c96923a5fc87d863916; no voters:
I20260925 22:50:01.123421 21787 leader_election.cc:288] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:01.123495 21792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:01.123725 21792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [term 1 LEADER]: Becoming Leader. State: Replica: 1481e5a4cae84c96923a5fc87d863916, State: Running, Role: LEADER
I20260925 22:50:01.124090 21792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1481e5a4cae84c96923a5fc87d863916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41329 } }
I20260925 22:50:01.124758 21787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:01.125267 21792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1481e5a4cae84c96923a5fc87d863916. Latest consensus state: current_term: 1 leader_uuid: "1481e5a4cae84c96923a5fc87d863916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1481e5a4cae84c96923a5fc87d863916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41329 } } }
I20260925 22:50:01.125383 21792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:01.125666 21793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1481e5a4cae84c96923a5fc87d863916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1481e5a4cae84c96923a5fc87d863916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41329 } } }
I20260925 22:50:01.125831 21793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:01.128006 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.113004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:36321 (local address 127.0.0.1:41329)
0925 22:50:01.113470 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:01.113486 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:01.113527 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:50:01.114268 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:01.114293 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:01.114702 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:01.114983 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:01.115933 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.116979 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.118175 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.118652 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.118763 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.121971 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:01.121977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:01.121990 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:01.122019 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:01.124000 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:01.124556 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:01.124561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:01.124567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:01.124684 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:01.125106 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:01.125110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:01.125111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:01.125696 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:01.126506 (+ 810us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:01.126707 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 22:50:01.126928 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
W20260925 22:50:01.129774 21807 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:50:01.129902 21807 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:50:01.130025 21796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:01.130878 21796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:01.131443 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:01.136713 21796 catalog_manager.cc:1347] Generated new cluster ID: 011fa787466c45759e4e68e66935836d
I20260925 22:50:01.136770 21796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:01.142802 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.133941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:42271 (local address 127.0.0.1:41329)
0925 22:50:01.134048 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:01.134052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:01.134139 (+ 87us) server_negotiation.cc:407] Connection header received
0925 22:50:01.134304 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:01.134308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:01.134350 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:01.134441 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:01.135241 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.136011 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.137196 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.137539 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.137635 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.139332 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:01.139336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:01.139339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:01.139361 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:01.141008 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:01.141559 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:01.141562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:01.141564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:01.141644 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:01.141969 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:01.141972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:01.141974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:01.142161 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:01.142285 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:01.142565 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 22:50:01.142651 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 22:50:01.162772 21796 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:01.164135 21796 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:01.178790 21796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1481e5a4cae84c96923a5fc87d863916: Generated new TSK 0
I20260925 22:50:01.179641 21796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:01.189253 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 21719
W20260925 22:50:01.194814 21704 connection.cc:535] client connection to 127.0.0.1:41329 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41329): Connection reset by peer (error 104)
2026-09-25T22:50:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:01 dist-test-slave-vdtf krb5kdc[21818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:01 dist-test-slave-vdtf krb5kdc[21818](info): set up 2 sockets
Sep 25 22:50:01 dist-test-slave-vdtf krb5kdc[21818](info): commencing operation
krb5kdc: starting...
W20260925 22:50:02.227859 806 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:02 dist-test-slave-vdtf krb5kdc[21818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:50:02.347385 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:02.458777 21831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:02.459089 21831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:02.459161 21831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:02.463160 21831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:02.463256 21831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:02.463296 21831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:02.463335 21831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:02.463370 21831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:02.468587 21831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:02.469607 21831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:02.470609 21831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:02.479102 21837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:02.479089 21831 server_base.cc:1034] running on GCE node
W20260925 22:50:02.479115 21839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:02.479074 21836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:02.479980 21831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:02.480988 21831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:02.482170 21831 hybrid_clock.cc:648] HybridClock initialized: now 1790376602482115 us; error 50 us; skew 500 ppm
Sep 25 22:50:02 dist-test-slave-vdtf krb5kdc[21818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:02.492265 21831 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:50:02.493182 21831 webserver.cc:460] Webserver started at http://127.0.0.1:40639/ using document root <none> and password file <none>
I20260925 22:50:02.493733 21831 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:02.493824 21831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:02.494076 21831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:02.496162 21831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "692b0dfeb3e04b028e7650aa27462105"
format_stamp: "Formatted at 2026-09-25 22:50:02 on dist-test-slave-vdtf"
server_key: "b60ab8f401485403c1aa920fa56bc63a"
server_key_iv: "e4485783a5c29dae9b866528c61684d6"
server_key_version: "encryptionkey@0"
I20260925 22:50:02.496699 21831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "692b0dfeb3e04b028e7650aa27462105"
format_stamp: "Formatted at 2026-09-25 22:50:02 on dist-test-slave-vdtf"
server_key: "b60ab8f401485403c1aa920fa56bc63a"
server_key_iv: "e4485783a5c29dae9b866528c61684d6"
server_key_version: "encryptionkey@0"
I20260925 22:50:02.500259 21831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:50:02.502496 21846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:02.503929 21831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:02.504069 21831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "692b0dfeb3e04b028e7650aa27462105"
format_stamp: "Formatted at 2026-09-25 22:50:02 on dist-test-slave-vdtf"
server_key: "b60ab8f401485403c1aa920fa56bc63a"
server_key_iv: "e4485783a5c29dae9b866528c61684d6"
server_key_version: "encryptionkey@0"
I20260925 22:50:02.504184 21831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:02.516302 21831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:02.519034 21831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:02.519229 21831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:02.526021 21831 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46025
I20260925 22:50:02.526089 21898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46025 every 8 connection(s)
I20260925 22:50:02.526958 21831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:02.529649 21899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:02.533778 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 21831
I20260925 22:50:02.533921 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:02.534195 806 external_mini_cluster.cc:1428] Setting key 9c2092de2b627e29eb80b8258f41ec10
I20260925 22:50:02.535346 21899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105: Bootstrap starting.
I20260925 22:50:02.538097 21899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:02.538962 21899 log.cc:824] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:02.540932 21899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105: No bootstrap required, opened a new log
I20260925 22:50:02.544124 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:02.535531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:36691 (local address 127.0.0.1:46025)
0925 22:50:02.535955 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:02.535973 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:02.536014 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:50:02.536858 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:02.536881 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:02.537305 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:02.537651 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:02.538067 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:02.539516 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:02.540458 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:02.541132 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:02.541298 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:02.542133 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:02.542139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:02.542154 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:02.542633 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:02.542697 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:02.542725 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:02.542859 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:02.542988 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 22:50:02.543126 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":151,"threads_started":1}
I20260925 22:50:02.545480 21899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692b0dfeb3e04b028e7650aa27462105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } }
I20260925 22:50:02.545692 21899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:02.545750 21899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692b0dfeb3e04b028e7650aa27462105, State: Initialized, Role: FOLLOWER
I20260925 22:50:02.546222 21899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692b0dfeb3e04b028e7650aa27462105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } }
I20260925 22:50:02.546371 21899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:02.546442 21899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:02.546535 21899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:02.547062 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:02.547499 21899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692b0dfeb3e04b028e7650aa27462105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } }
I20260925 22:50:02.547896 21899 leader_election.cc:302] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 692b0dfeb3e04b028e7650aa27462105; no voters:
I20260925 22:50:02.548224 21899 leader_election.cc:288] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:02.548290 21904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:02.548484 21904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [term 1 LEADER]: Becoming Leader. State: Replica: 692b0dfeb3e04b028e7650aa27462105, State: Running, Role: LEADER
I20260925 22:50:02.548789 21904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692b0dfeb3e04b028e7650aa27462105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } }
I20260925 22:50:02.549528 21899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:02.549968 21904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 692b0dfeb3e04b028e7650aa27462105. Latest consensus state: current_term: 1 leader_uuid: "692b0dfeb3e04b028e7650aa27462105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692b0dfeb3e04b028e7650aa27462105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } } }
I20260925 22:50:02.550107 21904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:02.550341 21904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "692b0dfeb3e04b028e7650aa27462105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692b0dfeb3e04b028e7650aa27462105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } } }
I20260925 22:50:02.550499 21904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:02.550806 21913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:02.553712 21913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:02.558717 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:02.552100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:50193 (local address 127.0.0.1:46025)
0925 22:50:02.552217 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:02.552221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:02.554102 (+ 1881us) server_negotiation.cc:407] Connection header received
0925 22:50:02.554298 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:02.554301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:02.554348 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:02.554463 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:02.554752 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:02.555276 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:02.556412 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:02.556743 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:02.556866 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:02.558090 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:02.558095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:02.558098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:02.558112 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:02.558141 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:02.558154 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:02.558238 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:02.558557 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 22:50:02.558592 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:50:02.559522 21913 catalog_manager.cc:1347] Generated new cluster ID: dc576eb595a945c3803b6ea0a1508cea
I20260925 22:50:02.559630 21913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:02.585538 21913 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:02.586521 21913 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:02.600405 21913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 692b0dfeb3e04b028e7650aa27462105: Generated new TSK 0
I20260925 22:50:02.601145 21913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:02.605394 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 21831
W20260925 22:50:02.610994 21816 connection.cc:535] client connection to 127.0.0.1:46025 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46025): Connection reset by peer (error 104)
2026-09-25T22:50:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:02 dist-test-slave-vdtf krb5kdc[21930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:02 dist-test-slave-vdtf krb5kdc[21930](info): set up 2 sockets
Sep 25 22:50:02 dist-test-slave-vdtf krb5kdc[21930](info): commencing operation
krb5kdc: starting...
W20260925 22:50:03.644018 806 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:03 dist-test-slave-vdtf krb5kdc[21930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:50:03.763602 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:03.877612 21943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:03.877880 21943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:03.877956 21943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:03.882143 21943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:03.882210 21943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:03.882228 21943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:03.882246 21943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:03.882261 21943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:03.887392 21943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:03.888437 21943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:03.889456 21943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:03.897775 21949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:03.897780 21951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:03.897807 21948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:03.897790 21943 server_base.cc:1034] running on GCE node
I20260925 22:50:03.898536 21943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:03.899566 21943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:03.900761 21943 hybrid_clock.cc:648] HybridClock initialized: now 1790376603900742 us; error 30 us; skew 500 ppm
Sep 25 22:50:03 dist-test-slave-vdtf krb5kdc[21930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:03.903488 21943 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:50:03.904493 21943 webserver.cc:460] Webserver started at http://127.0.0.1:40609/ using document root <none> and password file <none>
I20260925 22:50:03.905056 21943 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:03.905149 21943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:03.905381 21943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:03.907392 21943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "b55e751a7ad54c62b3e6b2207dbec7af"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-vdtf"
server_key: "25f4f94d767327b1046737204b0461ce"
server_key_iv: "b109e411cce9241f7d0123c6558c947e"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.907992 21943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "b55e751a7ad54c62b3e6b2207dbec7af"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-vdtf"
server_key: "25f4f94d767327b1046737204b0461ce"
server_key_iv: "b109e411cce9241f7d0123c6558c947e"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.911759 21943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:50:03.914232 21958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:03.915553 21943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:03.915750 21943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "b55e751a7ad54c62b3e6b2207dbec7af"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-vdtf"
server_key: "25f4f94d767327b1046737204b0461ce"
server_key_iv: "b109e411cce9241f7d0123c6558c947e"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.915871 21943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:03.920320 21943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:03.923208 21943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:03.923398 21943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:03.930236 21943 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44569
I20260925 22:50:03.930287 22010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44569 every 8 connection(s)
I20260925 22:50:03.931190 21943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:03.933964 22011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:03.939286 22011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af: Bootstrap starting.
I20260925 22:50:03.939636 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 21943
I20260925 22:50:03.939812 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:03.940119 806 external_mini_cluster.cc:1428] Setting key 0fded3675c590d9b2e4d1d0a612e4be4
I20260925 22:50:03.942028 22011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:03.942759 22011 log.cc:824] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:03.945029 22011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af: No bootstrap required, opened a new log
I20260925 22:50:03.945474 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.941452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:33853 (local address 127.0.0.1:44569)
0925 22:50:03.941878 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:03.941892 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:03.941928 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:50:03.942602 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:03.942619 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:03.942942 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:03.943282 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:03.944023 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:03.944029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:03.944043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:03.944054 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:03.944123 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:03.944148 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:03.944265 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:03.944417 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 22:50:03.944569 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260925 22:50:03.948256 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:03.949167 22011 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55e751a7ad54c62b3e6b2207dbec7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44569 } }
I20260925 22:50:03.949409 22011 raft_consensus.cc:374] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:03.949481 22011 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55e751a7ad54c62b3e6b2207dbec7af, State: Initialized, Role: FOLLOWER
I20260925 22:50:03.949944 22011 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55e751a7ad54c62b3e6b2207dbec7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44569 } }
I20260925 22:50:03.950098 22011 raft_consensus.cc:388] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:03.950186 22011 raft_consensus.cc:482] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:03.950302 22011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:03.951480 22011 raft_consensus.cc:504] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55e751a7ad54c62b3e6b2207dbec7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44569 } }
I20260925 22:50:03.951900 22011 leader_election.cc:302] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b55e751a7ad54c62b3e6b2207dbec7af; no voters:
I20260925 22:50:03.952008 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.950301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:48097 (local address 127.0.0.1:44569)
0925 22:50:03.950415 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:03.950420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:03.950718 (+ 298us) server_negotiation.cc:407] Connection header received
0925 22:50:03.950793 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:03.950796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:03.950828 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:03.950919 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:03.951605 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:03.951609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:03.951612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:03.951624 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:03.951651 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:03.951663 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:03.951769 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:03.951825 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 22:50:03.951863 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 22:50:03.952246 22011 leader_election.cc:288] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:03.952350 22021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:03.952544 22021 raft_consensus.cc:686] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [term 1 LEADER]: Becoming Leader. State: Replica: b55e751a7ad54c62b3e6b2207dbec7af, State: Running, Role: LEADER
I20260925 22:50:03.952877 22021 consensus_queue.cc:237] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55e751a7ad54c62b3e6b2207dbec7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44569 } }
I20260925 22:50:03.953459 22011 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:03.953972 22021 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55e751a7ad54c62b3e6b2207dbec7af. Latest consensus state: current_term: 1 leader_uuid: "b55e751a7ad54c62b3e6b2207dbec7af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55e751a7ad54c62b3e6b2207dbec7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44569 } } }
I20260925 22:50:03.954142 22021 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:03.954463 22022 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b55e751a7ad54c62b3e6b2207dbec7af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55e751a7ad54c62b3e6b2207dbec7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44569 } } }
I20260925 22:50:03.954555 22027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:03.955374 22022 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:03.957892 22027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:03.963362 22027 catalog_manager.cc:1347] Generated new cluster ID: b7e425ed4de04065a2643a9752a2e742
I20260925 22:50:03.963485 22027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:03.993456 22027 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:03.993798 22027 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:04.005263 22027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b55e751a7ad54c62b3e6b2207dbec7af: Generated new TSK 0
I20260925 22:50:04.005972 22027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:04.009035 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 21943
2026-09-25T22:50:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:04 dist-test-slave-vdtf krb5kdc[22042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:04 dist-test-slave-vdtf krb5kdc[22042](info): set up 2 sockets
Sep 25 22:50:04 dist-test-slave-vdtf krb5kdc[22042](info): commencing operation
krb5kdc: starting...
W20260925 22:50:05.071959 806 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:05 dist-test-slave-vdtf krb5kdc[22042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:50:05.191015 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:05.299345 22055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:05.299583 22055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:05.299623 22055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:05.303546 22055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:05.303606 22055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:05.303627 22055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:05.303656 22055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:05.303671 22055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:05.308928 22055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:05.309863 22055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:05.310834 22055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:05.319003 22060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:05.319044 22063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:05.319011 22061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:05.319345 22055 server_base.cc:1034] running on GCE node
I20260925 22:50:05.319978 22055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:05.321121 22055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:05.322340 22055 hybrid_clock.cc:648] HybridClock initialized: now 1790376605322309 us; error 48 us; skew 500 ppm
Sep 25 22:50:05 dist-test-slave-vdtf krb5kdc[22042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:05.325155 22055 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:50:05.326030 22055 webserver.cc:460] Webserver started at http://127.0.0.1:45405/ using document root <none> and password file <none>
I20260925 22:50:05.326642 22055 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:05.326732 22055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:05.326959 22055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:05.329109 22055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "e30f3182fb3e442bb070f37b8fcc17bb"
format_stamp: "Formatted at 2026-09-25 22:50:05 on dist-test-slave-vdtf"
server_key: "1c546738fa153f7d85304941206f3e27"
server_key_iv: "e63b4bc7d1236b71cefe864cd1c5bd9d"
server_key_version: "encryptionkey@0"
I20260925 22:50:05.329710 22055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "e30f3182fb3e442bb070f37b8fcc17bb"
format_stamp: "Formatted at 2026-09-25 22:50:05 on dist-test-slave-vdtf"
server_key: "1c546738fa153f7d85304941206f3e27"
server_key_iv: "e63b4bc7d1236b71cefe864cd1c5bd9d"
server_key_version: "encryptionkey@0"
I20260925 22:50:05.333315 22055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:50:05.335594 22070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:05.336869 22055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:05.337026 22055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "e30f3182fb3e442bb070f37b8fcc17bb"
format_stamp: "Formatted at 2026-09-25 22:50:05 on dist-test-slave-vdtf"
server_key: "1c546738fa153f7d85304941206f3e27"
server_key_iv: "e63b4bc7d1236b71cefe864cd1c5bd9d"
server_key_version: "encryptionkey@0"
I20260925 22:50:05.337141 22055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:05.359314 22055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:05.362172 22055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:05.362394 22055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:05.369769 22055 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40987
I20260925 22:50:05.369781 22122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40987 every 8 connection(s)
I20260925 22:50:05.370726 22055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:05.373529 22123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:05.377251 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 22055
I20260925 22:50:05.377365 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:05.377682 806 external_mini_cluster.cc:1428] Setting key 367e4d12d03f1557af1a636b0a45140d
I20260925 22:50:05.379724 22123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb: Bootstrap starting.
I20260925 22:50:05.382455 22123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:05.383289 22123 log.cc:824] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:05.385363 22123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb: No bootstrap required, opened a new log
Sep 25 22:50:05 dist-test-slave-vdtf krb5kdc[22042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:50:05.389101 22123 raft_consensus.cc:348] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3182fb3e442bb070f37b8fcc17bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40987 } }
I20260925 22:50:05.389314 22123 raft_consensus.cc:374] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:05.389376 22123 raft_consensus.cc:729] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30f3182fb3e442bb070f37b8fcc17bb, State: Initialized, Role: FOLLOWER
I20260925 22:50:05.389792 22123 consensus_queue.cc:260] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3182fb3e442bb070f37b8fcc17bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40987 } }
I20260925 22:50:05.389909 22123 raft_consensus.cc:388] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:05.390089 22123 raft_consensus.cc:482] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:05.390211 22123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:05.391034 22123 raft_consensus.cc:504] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3182fb3e442bb070f37b8fcc17bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40987 } }
I20260925 22:50:05.391376 22123 leader_election.cc:302] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e30f3182fb3e442bb070f37b8fcc17bb; no voters:
I20260925 22:50:05.391723 22123 leader_election.cc:288] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:05.391808 22128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:05.392081 22128 raft_consensus.cc:686] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [term 1 LEADER]: Becoming Leader. State: Replica: e30f3182fb3e442bb070f37b8fcc17bb, State: Running, Role: LEADER
I20260925 22:50:05.392406 22128 consensus_queue.cc:237] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3182fb3e442bb070f37b8fcc17bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40987 } }
I20260925 22:50:05.392987 22123 sys_catalog.cc:564] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:05.393110 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:05.379006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:56269 (local address 127.0.0.1:40987)
0925 22:50:05.379413 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:05.379432 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:05.379474 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:50:05.380311 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:05.380334 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:05.380773 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:05.381166 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:05.381834 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:05.383348 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:05.384374 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:05.385025 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:05.385201 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:05.387913 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:05.387918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:05.387930 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:05.387955 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:05.389857 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:05.390461 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:05.390467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:05.390479 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:05.390588 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:05.390922 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:05.390925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:05.390927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:05.391359 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:05.391582 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:05.391935 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 22:50:05.392181 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260925 22:50:05.394085 22129 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e30f3182fb3e442bb070f37b8fcc17bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3182fb3e442bb070f37b8fcc17bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40987 } } }
I20260925 22:50:05.394280 22129 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:05.394701 22130 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30f3182fb3e442bb070f37b8fcc17bb. Latest consensus state: current_term: 1 leader_uuid: "e30f3182fb3e442bb070f37b8fcc17bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30f3182fb3e442bb070f37b8fcc17bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40987 } } }
I20260925 22:50:05.394869 22130 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:05.395334 22137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:05.396447 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:05.398669 22137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:05.404798 22137 catalog_manager.cc:1347] Generated new cluster ID: 0212b79504984a45954e2fd5817bab1e
I20260925 22:50:05.404865 22137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:05.407433 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:05.398610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:37623 (local address 127.0.0.1:40987)
0925 22:50:05.398727 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:05.398733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:05.398957 (+ 224us) server_negotiation.cc:407] Connection header received
0925 22:50:05.399076 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:05.399079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:05.399126 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:05.399225 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:05.400060 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:05.400765 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:05.401581 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:05.401820 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:05.401936 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:05.403540 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:05.403544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:05.403546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:05.403574 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:05.405319 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:05.405903 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:05.405907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:05.405909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:05.405992 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:05.406405 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:05.406409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:05.406411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:05.406614 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:05.406735 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:05.407171 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 22:50:05.407272 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:50:05.413106 22137 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:05.413971 22137 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:05.420506 22137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e30f3182fb3e442bb070f37b8fcc17bb: Generated new TSK 0
I20260925 22:50:05.421203 22137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:05.426505 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 22055
W20260925 22:50:05.431937 22040 connection.cc:535] client connection to 127.0.0.1:40987 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40987): Connection reset by peer (error 104)
2026-09-25T22:50:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:05 dist-test-slave-vdtf krb5kdc[22154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:05 dist-test-slave-vdtf krb5kdc[22154](info): set up 2 sockets
Sep 25 22:50:05 dist-test-slave-vdtf krb5kdc[22154](info): commencing operation
krb5kdc: starting...
W20260925 22:50:06.459915 806 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:06 dist-test-slave-vdtf krb5kdc[22154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:50:06.576962 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:06.685261 22167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:06.685506 22167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:06.685546 22167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:06.689569 22167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:06.689622 22167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:06.689641 22167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:06.689657 22167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:06.689672 22167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:06.694722 22167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34349
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:06.695751 22167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:06.696718 22167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:06.705003 22172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:06.705009 22175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:06.705068 22167 server_base.cc:1034] running on GCE node
W20260925 22:50:06.705020 22173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:06.705754 22167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:06.706844 22167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:06.708029 22167 hybrid_clock.cc:648] HybridClock initialized: now 1790376606708009 us; error 32 us; skew 500 ppm
Sep 25 22:50:06 dist-test-slave-vdtf krb5kdc[22154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:06.710745 22167 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:50:06.711649 22167 webserver.cc:460] Webserver started at http://127.0.0.1:46323/ using document root <none> and password file <none>
I20260925 22:50:06.712276 22167 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:06.712368 22167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:06.712594 22167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:06.714584 22167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "4d587ebbb05b4c099686a9d1d0cd9727"
format_stamp: "Formatted at 2026-09-25 22:50:06 on dist-test-slave-vdtf"
server_key: "2d42c25b504377594441861f0b9bf772"
server_key_iv: "14c7159c643a1ad4cb06aa6b75a97d90"
server_key_version: "encryptionkey@0"
I20260925 22:50:06.715160 22167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "4d587ebbb05b4c099686a9d1d0cd9727"
format_stamp: "Formatted at 2026-09-25 22:50:06 on dist-test-slave-vdtf"
server_key: "2d42c25b504377594441861f0b9bf772"
server_key_iv: "14c7159c643a1ad4cb06aa6b75a97d90"
server_key_version: "encryptionkey@0"
I20260925 22:50:06.718833 22167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 22:50:06.721328 22182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:06.722486 22167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:06.722632 22167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "4d587ebbb05b4c099686a9d1d0cd9727"
format_stamp: "Formatted at 2026-09-25 22:50:06 on dist-test-slave-vdtf"
server_key: "2d42c25b504377594441861f0b9bf772"
server_key_iv: "14c7159c643a1ad4cb06aa6b75a97d90"
server_key_version: "encryptionkey@0"
I20260925 22:50:06.722756 22167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:06.751061 22167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:06.754106 22167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:06.754317 22167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:06.761416 22167 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34349
I20260925 22:50:06.761435 22234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34349 every 8 connection(s)
I20260925 22:50:06.762413 22167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:06.762782 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 22167
I20260925 22:50:06.762930 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:06.763230 806 external_mini_cluster.cc:1428] Setting key 0768e8717a695d736e6bac3521b1dd58
I20260925 22:50:06.765604 22235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:06.771946 22235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727: Bootstrap starting.
I20260925 22:50:06.772317 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:06.764713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:36621 (local address 127.0.0.1:34349)
0925 22:50:06.765130 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:06.765150 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:06.765191 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:50:06.765882 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:06.765900 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:06.766254 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:06.766522 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:06.766857 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:06.767895 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:06.769123 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:06.769576 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:06.769692 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:06.770585 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:06.770590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:06.770602 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:06.770966 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:06.771017 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:06.771037 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:06.771134 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:06.771373 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 22:50:06.771518 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":139,"threads_started":1}
I20260925 22:50:06.774361 22235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:06.775219 22235 log.cc:824] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:06.775255 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:06.777112 22235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727: No bootstrap required, opened a new log
I20260925 22:50:06.781343 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:06.777287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:60503 (local address 127.0.0.1:34349)
0925 22:50:06.777405 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:06.777409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:06.777570 (+ 161us) server_negotiation.cc:407] Connection header received
0925 22:50:06.777648 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:06.777651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:06.777702 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:06.777786 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:06.778008 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:06.778767 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:06.779551 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:06.779923 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:06.780034 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:06.780923 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:06.780927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:06.780930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:06.780943 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:06.780970 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:06.780982 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:06.781072 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:06.781188 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 22:50:06.781223 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:50:06.781379 22235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d587ebbb05b4c099686a9d1d0cd9727" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34349 } }
I20260925 22:50:06.781651 22235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:06.781749 22235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d587ebbb05b4c099686a9d1d0cd9727, State: Initialized, Role: FOLLOWER
I20260925 22:50:06.782217 22235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d587ebbb05b4c099686a9d1d0cd9727" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34349 } }
I20260925 22:50:06.782397 22235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:06.782473 22235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:06.782562 22235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:06.783363 22235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d587ebbb05b4c099686a9d1d0cd9727" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34349 } }
I20260925 22:50:06.783640 22235 leader_election.cc:302] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d587ebbb05b4c099686a9d1d0cd9727; no voters:
I20260925 22:50:06.783977 22235 leader_election.cc:288] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:06.784082 22245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:06.784340 22245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [term 1 LEADER]: Becoming Leader. State: Replica: 4d587ebbb05b4c099686a9d1d0cd9727, State: Running, Role: LEADER
I20260925 22:50:06.784708 22245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d587ebbb05b4c099686a9d1d0cd9727" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34349 } }
I20260925 22:50:06.785138 22235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:06.786473 22246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d587ebbb05b4c099686a9d1d0cd9727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d587ebbb05b4c099686a9d1d0cd9727" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34349 } } }
I20260925 22:50:06.786523 22247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d587ebbb05b4c099686a9d1d0cd9727. Latest consensus state: current_term: 1 leader_uuid: "4d587ebbb05b4c099686a9d1d0cd9727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d587ebbb05b4c099686a9d1d0cd9727" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34349 } } }
I20260925 22:50:06.786638 22247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:06.786623 22246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:06.787097 22256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:06.789878 22256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:06.795130 22256 catalog_manager.cc:1347] Generated new cluster ID: 59af32433c9c449abb4527ad8bba44fc
I20260925 22:50:06.795194 22256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:06.803390 22256 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:06.804317 22256 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:06.811997 22256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d587ebbb05b4c099686a9d1d0cd9727: Generated new TSK 0
I20260925 22:50:06.812698 22256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:06.816783 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 22167
W20260925 22:50:06.822062 22152 connection.cc:535] client connection to 127.0.0.1:34349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34349): Connection reset by peer (error 104)
2026-09-25T22:50:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:06 dist-test-slave-vdtf krb5kdc[22266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:06 dist-test-slave-vdtf krb5kdc[22266](info): set up 2 sockets
Sep 25 22:50:06 dist-test-slave-vdtf krb5kdc[22266](info): commencing operation
krb5kdc: starting...
W20260925 22:50:07.863952 806 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:07 dist-test-slave-vdtf krb5kdc[22266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:50:07.982863 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:08.092366 22279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:08.092680 22279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:08.092772 22279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:08.096755 22279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:08.096849 22279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:08.096889 22279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:08.096926 22279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:08.096958 22279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:08.102197 22279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:08.103302 22279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:08.104405 22279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:08.112782 22287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:08.112788 22284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:08.112788 22285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:08.113095 22279 server_base.cc:1034] running on GCE node
I20260925 22:50:08.113668 22279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:08.114794 22279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:08.115978 22279 hybrid_clock.cc:648] HybridClock initialized: now 1790376608115959 us; error 34 us; skew 500 ppm
Sep 25 22:50:08 dist-test-slave-vdtf krb5kdc[22266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:08.124728 22279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:50:08.125660 22279 webserver.cc:460] Webserver started at http://127.0.0.1:34201/ using document root <none> and password file <none>
I20260925 22:50:08.126252 22279 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:08.126346 22279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:08.126586 22279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:08.128729 22279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "368c2a2d04784d33a7f10b65ffd2793c"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-vdtf"
server_key: "f4f33738ddc2b267e702a6711ec08646"
server_key_iv: "e24911c7a7c18682d63ed0793e1bf56d"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.129325 22279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "368c2a2d04784d33a7f10b65ffd2793c"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-vdtf"
server_key: "f4f33738ddc2b267e702a6711ec08646"
server_key_iv: "e24911c7a7c18682d63ed0793e1bf56d"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.133086 22279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:50:08.135576 22294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.136883 22279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:08.137032 22279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "368c2a2d04784d33a7f10b65ffd2793c"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-vdtf"
server_key: "f4f33738ddc2b267e702a6711ec08646"
server_key_iv: "e24911c7a7c18682d63ed0793e1bf56d"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.137156 22279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:08.141803 22279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:08.145140 22279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:08.145344 22279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:08.152493 22279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40573
I20260925 22:50:08.152520 22346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40573 every 8 connection(s)
I20260925 22:50:08.153427 22279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:08.156152 22347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:08.159008 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 22279
I20260925 22:50:08.159158 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:08.159447 806 external_mini_cluster.cc:1428] Setting key ded91d12f7e8984dcd288c5b34eaac6c
I20260925 22:50:08.162127 22347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c: Bootstrap starting.
I20260925 22:50:08.164700 22347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:08.164920 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:08.160996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:41025 (local address 127.0.0.1:40573)
0925 22:50:08.161417 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:08.161435 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:08.161483 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:50:08.162101 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:08.162118 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:08.162469 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:08.162753 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:08.163601 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:08.163606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:08.163618 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:08.163629 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:08.163727 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:08.163762 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:08.163873 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:08.163986 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 22:50:08.164165 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":153,"threads_started":1}
I20260925 22:50:08.165489 22347 log.cc:824] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:08.167599 22347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c: No bootstrap required, opened a new log
I20260925 22:50:08.167917 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:08.171013 22347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368c2a2d04784d33a7f10b65ffd2793c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } }
I20260925 22:50:08.171329 22347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:08.171412 22347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 368c2a2d04784d33a7f10b65ffd2793c, State: Initialized, Role: FOLLOWER
I20260925 22:50:08.171823 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:08.170343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:50509 (local address 127.0.0.1:40573)
0925 22:50:08.170460 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:08.170463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:08.170474 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:50:08.170505 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:08.170507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:08.170532 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:08.170597 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:08.171327 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:08.171334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:08.171337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:08.171351 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:08.171384 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:08.171395 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:08.171493 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:08.171618 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 22:50:08.171661 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:50:08.171944 22347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368c2a2d04784d33a7f10b65ffd2793c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } }
I20260925 22:50:08.172116 22347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:08.172187 22347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:08.172286 22347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:08.173487 22347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368c2a2d04784d33a7f10b65ffd2793c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } }
I20260925 22:50:08.173844 22347 leader_election.cc:302] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 368c2a2d04784d33a7f10b65ffd2793c; no voters:
I20260925 22:50:08.174180 22347 leader_election.cc:288] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:08.174228 22357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:08.174563 22357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [term 1 LEADER]: Becoming Leader. State: Replica: 368c2a2d04784d33a7f10b65ffd2793c, State: Running, Role: LEADER
I20260925 22:50:08.174862 22357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368c2a2d04784d33a7f10b65ffd2793c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } }
I20260925 22:50:08.175416 22347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:08.176610 22359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 368c2a2d04784d33a7f10b65ffd2793c. Latest consensus state: current_term: 1 leader_uuid: "368c2a2d04784d33a7f10b65ffd2793c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368c2a2d04784d33a7f10b65ffd2793c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } } }
I20260925 22:50:08.176746 22359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:08.177026 22358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "368c2a2d04784d33a7f10b65ffd2793c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368c2a2d04784d33a7f10b65ffd2793c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40573 } } }
I20260925 22:50:08.177170 22358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:08.177114 22366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:08.180775 22366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:08.186110 22366 catalog_manager.cc:1347] Generated new cluster ID: 3f6dc53582ee4b7b8f2c135a7b86cfc6
I20260925 22:50:08.186204 22366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:08.193908 22366 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:08.194195 22366 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:08.199648 22366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 368c2a2d04784d33a7f10b65ffd2793c: Generated new TSK 0
I20260925 22:50:08.200381 22366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:08.210197 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 22279
2026-09-25T22:50:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:08 dist-test-slave-vdtf krb5kdc[22378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:08 dist-test-slave-vdtf krb5kdc[22378](info): set up 2 sockets
Sep 25 22:50:08 dist-test-slave-vdtf krb5kdc[22378](info): commencing operation
krb5kdc: starting...
W20260925 22:50:09.248003 806 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:09 dist-test-slave-vdtf krb5kdc[22378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:50:09.367539 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:44431
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:44431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:09.476022 22391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:09.476300 22391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:09.476368 22391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:09.480314 22391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:09.480387 22391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:09.480425 22391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:09.480476 22391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:09.480520 22391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:09.485630 22391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:44431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:09.486636 22391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:09.487636 22391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:09.495939 22399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:09.495960 22397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:09.495985 22391 server_base.cc:1034] running on GCE node
W20260925 22:50:09.495939 22396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:09.496796 22391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:09.497836 22391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:09.499014 22391 hybrid_clock.cc:648] HybridClock initialized: now 1790376609499002 us; error 25 us; skew 500 ppm
Sep 25 22:50:09 dist-test-slave-vdtf krb5kdc[22378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:09.501699 22391 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:50:09.502487 22391 webserver.cc:460] Webserver started at http://127.0.201.190:35473/ using document root <none> and password file <none>
I20260925 22:50:09.503018 22391 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:09.503103 22391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:09.503324 22391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:09.505316 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "2847e2e2c5954385b73d86ffab7ee599"
format_stamp: "Formatted at 2026-09-25 22:50:09 on dist-test-slave-vdtf"
server_key: "17527e16d7f60a35b5c4e6fbafe20153"
server_key_iv: "1ec64df52afeb69c7fed2bfd75ec40d8"
server_key_version: "encryptionkey@0"
I20260925 22:50:09.505869 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "2847e2e2c5954385b73d86ffab7ee599"
format_stamp: "Formatted at 2026-09-25 22:50:09 on dist-test-slave-vdtf"
server_key: "17527e16d7f60a35b5c4e6fbafe20153"
server_key_iv: "1ec64df52afeb69c7fed2bfd75ec40d8"
server_key_version: "encryptionkey@0"
I20260925 22:50:09.509506 22391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:50:09.511844 22406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:09.513039 22391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:09.513168 22391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "2847e2e2c5954385b73d86ffab7ee599"
format_stamp: "Formatted at 2026-09-25 22:50:09 on dist-test-slave-vdtf"
server_key: "17527e16d7f60a35b5c4e6fbafe20153"
server_key_iv: "1ec64df52afeb69c7fed2bfd75ec40d8"
server_key_version: "encryptionkey@0"
I20260925 22:50:09.513271 22391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:09.529464 22391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:09.532656 22391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:09.532846 22391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:09.539587 22391 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:44431
I20260925 22:50:09.539603 22458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:44431 every 8 connection(s)
I20260925 22:50:09.540473 22391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:09.542914 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 22391
I20260925 22:50:09.543068 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:09.543277 22459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:09.543383 806 external_mini_cluster.cc:1428] Setting key 3d78543cfddc201f9feeccd185c82b79
I20260925 22:50:09.550129 22459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599: Bootstrap starting.
Sep 25 22:50:09 dist-test-slave-vdtf krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:50:09.552901 22459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:09.553805 22459 log.cc:824] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:09.556007 22459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599: No bootstrap required, opened a new log
I20260925 22:50:09.558053 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.544679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39585 (local address 127.0.201.190:44431)
0925 22:50:09.545072 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.545094 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.545127 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:50:09.545929 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.545946 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.546286 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.546549 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.547444 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.548466 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.549637 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.550095 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.550206 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.552927 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:09.552933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:09.552950 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:09.552983 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:09.554979 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:09.555468 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:09.555473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:09.555479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:09.555608 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:09.555962 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:09.555965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:09.555967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:09.556467 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:09.556676 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:09.556897 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 22:50:09.557117 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":131,"threads_started":1}
I20260925 22:50:09.560225 22459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2847e2e2c5954385b73d86ffab7ee599" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44431 } }
I20260925 22:50:09.560495 22459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:09.560555 22459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2847e2e2c5954385b73d86ffab7ee599, State: Initialized, Role: FOLLOWER
I20260925 22:50:09.561003 22459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2847e2e2c5954385b73d86ffab7ee599" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44431 } }
I20260925 22:50:09.561120 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:09.561180 22459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:09.561244 22459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:09.561331 22459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:09.562426 22459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2847e2e2c5954385b73d86ffab7ee599" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44431 } }
I20260925 22:50:09.562805 22459 leader_election.cc:302] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2847e2e2c5954385b73d86ffab7ee599; no voters:
I20260925 22:50:09.563167 22459 leader_election.cc:288] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:09.563239 22468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:09.563479 22468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [term 1 LEADER]: Becoming Leader. State: Replica: 2847e2e2c5954385b73d86ffab7ee599, State: Running, Role: LEADER
I20260925 22:50:09.563838 22468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2847e2e2c5954385b73d86ffab7ee599" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44431 } }
I20260925 22:50:09.564478 22459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:09.565455 22470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2847e2e2c5954385b73d86ffab7ee599. Latest consensus state: current_term: 1 leader_uuid: "2847e2e2c5954385b73d86ffab7ee599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2847e2e2c5954385b73d86ffab7ee599" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44431 } } }
I20260925 22:50:09.565646 22470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:09.566026 22471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2847e2e2c5954385b73d86ffab7ee599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2847e2e2c5954385b73d86ffab7ee599" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 44431 } } }
I20260925 22:50:09.566094 22478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:09.566202 22471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:09.569541 22478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:09.572614 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.563279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40867 (local address 127.0.201.190:44431)
0925 22:50:09.563388 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.563392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.563911 (+ 519us) server_negotiation.cc:407] Connection header received
0925 22:50:09.564071 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.564074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.564121 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.564202 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.564890 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.565602 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.566682 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.567016 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.567115 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.568860 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:09.568864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:09.568867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:09.568890 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:09.570610 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:09.571160 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:09.571164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:09.571166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:09.571269 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:09.571643 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:09.571646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:09.571648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:09.571915 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:09.572065 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:09.572360 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 22:50:09.572460 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:50:09.575455 22478 catalog_manager.cc:1347] Generated new cluster ID: 593ff655adaa43a5b0d6e11aa686bf93
I20260925 22:50:09.575528 22478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:09.603471 22478 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:09.604426 22478 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:09.612289 22478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2847e2e2c5954385b73d86ffab7ee599: Generated new TSK 0
I20260925 22:50:09.612982 22478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:09.621896 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 22391
W20260925 22:50:09.627498 22376 connection.cc:535] client connection to 127.0.201.190:44431 recv error: Network error: failed to read from TLS socket (remote: 127.0.201.190:44431): Connection reset by peer (error 104)
2026-09-25T22:50:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:09 dist-test-slave-vdtf krb5kdc[22490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:09 dist-test-slave-vdtf krb5kdc[22490](info): set up 2 sockets
Sep 25 22:50:09 dist-test-slave-vdtf krb5kdc[22490](info): commencing operation
krb5kdc: starting...
W20260925 22:50:10.647969 806 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:10 dist-test-slave-vdtf krb5kdc[22490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:50:10.767097 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:43685
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:43685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:10.876852 22503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:10.877128 22503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:10.877197 22503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:10.881165 22503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:10.881239 22503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:10.881278 22503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:10.881330 22503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:10.881378 22503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:10.886561 22503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:43685
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:43685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:10.887558 22503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:10.888567 22503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:10.896682 22508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:10.896726 22509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:10.896775 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:10.897079 22503 server_base.cc:1034] running on GCE node
I20260925 22:50:10.897604 22503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:10.898636 22503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:10.899771 22503 hybrid_clock.cc:648] HybridClock initialized: now 1790376610899753 us; error 30 us; skew 500 ppm
Sep 25 22:50:10 dist-test-slave-vdtf krb5kdc[22490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:10.902539 22503 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:50:10.903386 22503 webserver.cc:460] Webserver started at http://127.0.201.190:43493/ using document root <none> and password file <none>
I20260925 22:50:10.903957 22503 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:10.904018 22503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:10.904181 22503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:10.906083 22503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "8b42ca71285543058bef6b6ee2d94f8a"
format_stamp: "Formatted at 2026-09-25 22:50:10 on dist-test-slave-vdtf"
server_key: "a5e144194c9df1a2a90ba9b52cab0926"
server_key_iv: "c275b0091912c9ca9f4c0ecba6478994"
server_key_version: "encryptionkey@0"
I20260925 22:50:10.906800 22503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "8b42ca71285543058bef6b6ee2d94f8a"
format_stamp: "Formatted at 2026-09-25 22:50:10 on dist-test-slave-vdtf"
server_key: "a5e144194c9df1a2a90ba9b52cab0926"
server_key_iv: "c275b0091912c9ca9f4c0ecba6478994"
server_key_version: "encryptionkey@0"
I20260925 22:50:10.910542 22503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 22:50:10.912998 22518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:10.914284 22503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:10.914402 22503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "8b42ca71285543058bef6b6ee2d94f8a"
format_stamp: "Formatted at 2026-09-25 22:50:10 on dist-test-slave-vdtf"
server_key: "a5e144194c9df1a2a90ba9b52cab0926"
server_key_iv: "c275b0091912c9ca9f4c0ecba6478994"
server_key_version: "encryptionkey@0"
I20260925 22:50:10.914498 22503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:10.934520 22503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:10.937480 22503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:10.937700 22503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:10.944916 22503 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:43685
I20260925 22:50:10.944972 22570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:43685 every 8 connection(s)
I20260925 22:50:10.945936 22503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:10.948779 22571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:10.953207 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 22503
I20260925 22:50:10.953357 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:10.953663 806 external_mini_cluster.cc:1428] Setting key 8fcb6e3366b7db888321839f0681230c
I20260925 22:50:10.954638 22571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a: Bootstrap starting.
I20260925 22:50:10.956940 22571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:10.957738 22571 log.cc:824] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:10.959499 22571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a: No bootstrap required, opened a new log
I20260925 22:50:10.962689 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:10.955092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54757 (local address 127.0.201.190:43685)
0925 22:50:10.955492 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:10.955507 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:10.955552 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:50:10.956225 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:10.956243 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:10.956583 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:10.956857 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:10.957288 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.958344 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.959250 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.959819 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.959986 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:10.960971 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:10.960976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:10.960989 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:10.961369 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:10.961424 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:10.961447 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:10.961608 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:10.961831 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 22:50:10.961971 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":131,"threads_started":1}
I20260925 22:50:10.963337 22571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b42ca71285543058bef6b6ee2d94f8a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43685 } }
I20260925 22:50:10.963582 22571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:10.963711 22571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b42ca71285543058bef6b6ee2d94f8a, State: Initialized, Role: FOLLOWER
I20260925 22:50:10.964174 22571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b42ca71285543058bef6b6ee2d94f8a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43685 } }
I20260925 22:50:10.964325 22571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:10.964386 22571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:10.964483 22571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:10.965520 22571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b42ca71285543058bef6b6ee2d94f8a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43685 } }
I20260925 22:50:10.965595 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:10.965855 22571 leader_election.cc:302] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b42ca71285543058bef6b6ee2d94f8a; no voters:
I20260925 22:50:10.966143 22571 leader_election.cc:288] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:10.966253 22576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:10.966514 22576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [term 1 LEADER]: Becoming Leader. State: Replica: 8b42ca71285543058bef6b6ee2d94f8a, State: Running, Role: LEADER
I20260925 22:50:10.966903 22576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b42ca71285543058bef6b6ee2d94f8a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43685 } }
I20260925 22:50:10.967377 22571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:10.968173 22577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b42ca71285543058bef6b6ee2d94f8a. Latest consensus state: current_term: 1 leader_uuid: "8b42ca71285543058bef6b6ee2d94f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b42ca71285543058bef6b6ee2d94f8a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43685 } } }
I20260925 22:50:10.968290 22577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:10.968662 22578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b42ca71285543058bef6b6ee2d94f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b42ca71285543058bef6b6ee2d94f8a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 43685 } } }
I20260925 22:50:10.968745 22578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a [sys.catalog]: This master's current role is: LEADER
W20260925 22:50:10.972687 22596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:50:10.972769 22596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:50:10.972836 22589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:10.973752 22589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:10.973930 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:10.968883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34473 (local address 127.0.201.190:43685)
0925 22:50:10.969027 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:10.969034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:10.969503 (+ 469us) server_negotiation.cc:407] Connection header received
0925 22:50:10.969673 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:10.969675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:10.969716 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:10.969775 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:10.970184 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.970683 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.971856 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.972183 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.972262 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:10.973360 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:10.973364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:10.973367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:10.973380 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:10.973409 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:10.973423 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:10.973520 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:10.973705 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 22:50:10.973742 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 22:50:10.979333 22589 catalog_manager.cc:1347] Generated new cluster ID: d294ecb9ac6d4d4cb621428e03baec7c
I20260925 22:50:10.979400 22589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:10.987875 22589 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:10.989305 22589 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:11.003139 22589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b42ca71285543058bef6b6ee2d94f8a: Generated new TSK 0
I20260925 22:50:11.004309 22589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:11.011137 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 22503
W20260925 22:50:11.020239 22488 connection.cc:535] client connection to 127.0.201.190:43685 recv error: Network error: failed to read from TLS socket (remote: 127.0.201.190:43685): Connection reset by peer (error 104)
2026-09-25T22:50:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:11 dist-test-slave-vdtf krb5kdc[22602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:11 dist-test-slave-vdtf krb5kdc[22602](info): set up 2 sockets
Sep 25 22:50:11 dist-test-slave-vdtf krb5kdc[22602](info): commencing operation
krb5kdc: starting...
W20260925 22:50:12.099987 806 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:12 dist-test-slave-vdtf krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:50:12.219439 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:35049
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:35049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:12.325862 22615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:12.326135 22615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:12.326210 22615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:12.330186 22615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:12.330266 22615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:12.330304 22615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:12.330356 22615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:12.330395 22615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:12.335475 22615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:35049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:35049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:12.336495 22615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:12.337482 22615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:12.345949 22623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:12.345971 22620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:12.345971 22621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:12.346062 22615 server_base.cc:1034] running on GCE node
I20260925 22:50:12.346791 22615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:12.347961 22615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:12.349164 22615 hybrid_clock.cc:648] HybridClock initialized: now 1790376612349147 us; error 26 us; skew 500 ppm
Sep 25 22:50:12 dist-test-slave-vdtf krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:12.351946 22615 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:50:12.352845 22615 webserver.cc:460] Webserver started at http://127.0.201.190:42841/ using document root <none> and password file <none>
I20260925 22:50:12.353387 22615 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:12.353473 22615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:12.353696 22615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:12.355790 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "c10606e85fa241a3afd25d4b7a8e5161"
format_stamp: "Formatted at 2026-09-25 22:50:12 on dist-test-slave-vdtf"
server_key: "4896d28809836c157963e1562f5596a3"
server_key_iv: "cba2a3e3e334be19048c8fbb938f5160"
server_key_version: "encryptionkey@0"
I20260925 22:50:12.356357 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "c10606e85fa241a3afd25d4b7a8e5161"
format_stamp: "Formatted at 2026-09-25 22:50:12 on dist-test-slave-vdtf"
server_key: "4896d28809836c157963e1562f5596a3"
server_key_iv: "cba2a3e3e334be19048c8fbb938f5160"
server_key_version: "encryptionkey@0"
I20260925 22:50:12.360033 22615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:50:12.362414 22630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:12.363643 22615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:12.363803 22615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "c10606e85fa241a3afd25d4b7a8e5161"
format_stamp: "Formatted at 2026-09-25 22:50:12 on dist-test-slave-vdtf"
server_key: "4896d28809836c157963e1562f5596a3"
server_key_iv: "cba2a3e3e334be19048c8fbb938f5160"
server_key_version: "encryptionkey@0"
I20260925 22:50:12.363914 22615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:12.368496 22615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:12.371870 22615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:12.372066 22615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:12.378858 22615 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:35049
I20260925 22:50:12.378870 22682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:35049 every 8 connection(s)
I20260925 22:50:12.379822 22615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:12.382506 22683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:12.384917 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 22615
I20260925 22:50:12.385062 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:12.385370 806 external_mini_cluster.cc:1428] Setting key 62bcf8a223a9463f5349cb7c057fbc89
I20260925 22:50:12.388783 22683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161: Bootstrap starting.
I20260925 22:50:12.390480 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:12.386695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44679 (local address 127.0.201.190:35049)
0925 22:50:12.387099 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:12.387113 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:12.387149 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:50:12.387860 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:12.387879 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:12.388168 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:12.388444 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:12.389290 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:12.389295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:12.389306 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:12.389316 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:12.389366 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:12.389384 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:12.389492 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:12.389612 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 22:50:12.389782 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":138,"threads_started":1}
I20260925 22:50:12.391319 22683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:12.392107 22683 log.cc:824] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:12.393350 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:12.394331 22683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161: No bootstrap required, opened a new log
I20260925 22:50:12.397406 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:12.395548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40173 (local address 127.0.201.190:35049)
0925 22:50:12.395697 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:12.395702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:12.395907 (+ 205us) server_negotiation.cc:407] Connection header received
0925 22:50:12.396025 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:12.396027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:12.396055 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:12.396125 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:12.396823 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:12.396826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:12.396828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:12.396837 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:12.396860 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:12.396869 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:12.396936 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:12.397219 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 22:50:12.397275 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:50:12.399067 22683 raft_consensus.cc:348] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10606e85fa241a3afd25d4b7a8e5161" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35049 } }
I20260925 22:50:12.399284 22683 raft_consensus.cc:374] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:12.399359 22683 raft_consensus.cc:729] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c10606e85fa241a3afd25d4b7a8e5161, State: Initialized, Role: FOLLOWER
I20260925 22:50:12.399894 22683 consensus_queue.cc:260] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10606e85fa241a3afd25d4b7a8e5161" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35049 } }
I20260925 22:50:12.400079 22683 raft_consensus.cc:388] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:12.400156 22683 raft_consensus.cc:482] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:12.400255 22683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:12.401162 22683 raft_consensus.cc:504] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10606e85fa241a3afd25d4b7a8e5161" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35049 } }
I20260925 22:50:12.401489 22683 leader_election.cc:302] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c10606e85fa241a3afd25d4b7a8e5161; no voters:
I20260925 22:50:12.401778 22683 leader_election.cc:288] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:12.401960 22693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:12.402233 22693 raft_consensus.cc:686] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [term 1 LEADER]: Becoming Leader. State: Replica: c10606e85fa241a3afd25d4b7a8e5161, State: Running, Role: LEADER
I20260925 22:50:12.402536 22693 consensus_queue.cc:237] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10606e85fa241a3afd25d4b7a8e5161" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35049 } }
I20260925 22:50:12.402959 22683 sys_catalog.cc:564] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:12.404142 22694 sys_catalog.cc:455] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c10606e85fa241a3afd25d4b7a8e5161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10606e85fa241a3afd25d4b7a8e5161" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35049 } } }
I20260925 22:50:12.404271 22694 sys_catalog.cc:458] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:12.404707 22702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:12.404659 22695 sys_catalog.cc:455] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c10606e85fa241a3afd25d4b7a8e5161. Latest consensus state: current_term: 1 leader_uuid: "c10606e85fa241a3afd25d4b7a8e5161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10606e85fa241a3afd25d4b7a8e5161" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 35049 } } }
I20260925 22:50:12.404821 22695 sys_catalog.cc:458] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:12.407631 22702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:12.412700 22702 catalog_manager.cc:1347] Generated new cluster ID: c80b48b8cab64821887c1443a19520ae
I20260925 22:50:12.412802 22702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:12.445549 22702 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:12.445951 22702 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:12.457305 22702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c10606e85fa241a3afd25d4b7a8e5161: Generated new TSK 0
I20260925 22:50:12.458204 22702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:12.472153 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 22615
2026-09-25T22:50:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:12 dist-test-slave-vdtf krb5kdc[22714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:12 dist-test-slave-vdtf krb5kdc[22714](info): set up 2 sockets
Sep 25 22:50:12 dist-test-slave-vdtf krb5kdc[22714](info): commencing operation
krb5kdc: starting...
W20260925 22:50:13.495942 806 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:13 dist-test-slave-vdtf krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:50:13.610424 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:36117
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:36117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:13.716367 22727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:13.716645 22727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:13.716713 22727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:13.720840 22727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:13.720920 22727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:13.720957 22727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:13.721009 22727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:13.721051 22727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:13.726166 22727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:36117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:36117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:13.727169 22727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:13.728195 22727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:13.736371 22732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:13.736372 22735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:13.736372 22733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:13.736452 22727 server_base.cc:1034] running on GCE node
I20260925 22:50:13.737279 22727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:13.738336 22727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:13.739518 22727 hybrid_clock.cc:648] HybridClock initialized: now 1790376613739505 us; error 28 us; skew 500 ppm
Sep 25 22:50:13 dist-test-slave-vdtf krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:13.746155 22727 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:50:13.746973 22727 webserver.cc:460] Webserver started at http://127.0.201.190:33889/ using document root <none> and password file <none>
I20260925 22:50:13.747483 22727 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:13.747537 22727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:13.747759 22727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:13.749727 22727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "45f181d59e8549368d9b79bd495cfbe4"
format_stamp: "Formatted at 2026-09-25 22:50:13 on dist-test-slave-vdtf"
server_key: "58f0f7cfc9fc263ac6651a6beecac4e2"
server_key_iv: "325a21bf2d05bf572dfe602427e19d18"
server_key_version: "encryptionkey@0"
I20260925 22:50:13.750221 22727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "45f181d59e8549368d9b79bd495cfbe4"
format_stamp: "Formatted at 2026-09-25 22:50:13 on dist-test-slave-vdtf"
server_key: "58f0f7cfc9fc263ac6651a6beecac4e2"
server_key_iv: "325a21bf2d05bf572dfe602427e19d18"
server_key_version: "encryptionkey@0"
I20260925 22:50:13.753750 22727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:50:13.756150 22742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:13.757356 22727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:13.757453 22727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "45f181d59e8549368d9b79bd495cfbe4"
format_stamp: "Formatted at 2026-09-25 22:50:13 on dist-test-slave-vdtf"
server_key: "58f0f7cfc9fc263ac6651a6beecac4e2"
server_key_iv: "325a21bf2d05bf572dfe602427e19d18"
server_key_version: "encryptionkey@0"
I20260925 22:50:13.757535 22727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:13.785209 22727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:13.788511 22727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:13.788717 22727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:13.795397 22727 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:36117
I20260925 22:50:13.795491 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:36117 every 8 connection(s)
I20260925 22:50:13.796577 22727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:13.799270 22795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:13.804832 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4: Bootstrap starting.
I20260925 22:50:13.806274 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 22727
I20260925 22:50:13.806429 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:13.806774 806 external_mini_cluster.cc:1428] Setting key 72dadde5e3d60c10ec4f3041c4e0eec8
I20260925 22:50:13.807410 22795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:13.808286 22795 log.cc:824] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:13.810374 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4: No bootstrap required, opened a new log
I20260925 22:50:13.814533 22795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f181d59e8549368d9b79bd495cfbe4" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 36117 } }
I20260925 22:50:13.814743 22795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:13.814802 22795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f181d59e8549368d9b79bd495cfbe4, State: Initialized, Role: FOLLOWER
I20260925 22:50:13.815239 22795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f181d59e8549368d9b79bd495cfbe4" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 36117 } }
I20260925 22:50:13.815356 22795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:13.815440 22795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:13.815562 22795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 22:50:13 dist-test-slave-vdtf krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:50:13.816804 22795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f181d59e8549368d9b79bd495cfbe4" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 36117 } }
I20260925 22:50:13.817164 22795 leader_election.cc:302] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f181d59e8549368d9b79bd495cfbe4; no voters:
I20260925 22:50:13.817499 22795 leader_election.cc:288] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:13.817610 22800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:13.817827 22800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [term 1 LEADER]: Becoming Leader. State: Replica: 45f181d59e8549368d9b79bd495cfbe4, State: Running, Role: LEADER
I20260925 22:50:13.818219 22800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f181d59e8549368d9b79bd495cfbe4" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 36117 } }
I20260925 22:50:13.818994 22795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:13.820124 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f181d59e8549368d9b79bd495cfbe4. Latest consensus state: current_term: 1 leader_uuid: "45f181d59e8549368d9b79bd495cfbe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f181d59e8549368d9b79bd495cfbe4" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 36117 } } }
I20260925 22:50:13.820154 22801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f181d59e8549368d9b79bd495cfbe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f181d59e8549368d9b79bd495cfbe4" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 36117 } } }
I20260925 22:50:13.820345 22801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:13.820616 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:13.820919 22809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:13.822801 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:13.807843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43005 (local address 127.0.201.190:36117)
0925 22:50:13.808314 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:13.808333 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:13.808375 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:50:13.809170 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:13.809193 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:13.809598 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:13.809939 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:13.810629 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.812140 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.813007 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.813799 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.813962 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:13.816977 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:13.816988 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:13.817001 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:13.817028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:13.819224 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:13.819964 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:13.819998 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:13.820008 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:13.820367 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:13.820729 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:13.820733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:13.820735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:13.821278 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:13.821559 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:13.821810 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 22:50:13.822097 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":151,"threads_started":1}
I20260925 22:50:13.824550 22809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:13.825747 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:13.830152 22809 catalog_manager.cc:1347] Generated new cluster ID: 27d61058db6342ea8bd8dbd6b82c6085
I20260925 22:50:13.830210 22809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:13.837497 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:13.827920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44727 (local address 127.0.201.190:36117)
0925 22:50:13.828018 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:13.828021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:13.828265 (+ 244us) server_negotiation.cc:407] Connection header received
0925 22:50:13.828371 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:13.828374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:13.828419 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:13.828501 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:13.829432 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.830633 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.831797 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.832146 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.832255 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:13.833754 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:13.833758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:13.833761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:13.833786 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:13.835430 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:13.835948 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:13.835952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:13.835954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:13.836033 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:13.836318 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:13.836322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:13.836324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:13.836523 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:13.836656 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:13.837233 (+ 577us) server_negotiation.cc:299] Negotiation successful
0925 22:50:13.837342 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:50:13.851816 22809 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:13.853039 22809 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:13.864405 22809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f181d59e8549368d9b79bd495cfbe4: Generated new TSK 0
I20260925 22:50:13.865127 22809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:13.876391 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 22727
2026-09-25T22:50:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:13 dist-test-slave-vdtf krb5kdc[22826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:13 dist-test-slave-vdtf krb5kdc[22826](info): set up 2 sockets
Sep 25 22:50:13 dist-test-slave-vdtf krb5kdc[22826](info): commencing operation
krb5kdc: starting...
W20260925 22:50:14.903872 806 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:14 dist-test-slave-vdtf krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:50:15.020733 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42079
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:39147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:42079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:15.128993 22839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:15.129313 22839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:15.129387 22839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:15.133467 22839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:15.133549 22839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:15.133587 22839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:15.133641 22839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:15.133679 22839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:15.138856 22839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:39147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:42079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:42079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:15.139988 22839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:15.140998 22839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:15.149107 22847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:15.149305 22839 server_base.cc:1034] running on GCE node
W20260925 22:50:15.149109 22845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:15.149108 22844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:15.149915 22839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:15.150954 22839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:15.152132 22839 hybrid_clock.cc:648] HybridClock initialized: now 1790376615152119 us; error 26 us; skew 500 ppm
Sep 25 22:50:15 dist-test-slave-vdtf krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:15.154852 22839 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:50:15.155791 22839 webserver.cc:460] Webserver started at http://127.0.201.190:45577/ using document root <none> and password file <none>
I20260925 22:50:15.156333 22839 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:15.156419 22839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:15.156641 22839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:15.158640 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "5bf9cac5cf03455390605dead837345c"
format_stamp: "Formatted at 2026-09-25 22:50:15 on dist-test-slave-vdtf"
server_key: "e0eb335d24b91dc420f9ed63c2ec7029"
server_key_iv: "d2b295a0f0fc10aa150c5c4f9557159a"
server_key_version: "encryptionkey@0"
I20260925 22:50:15.159195 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "5bf9cac5cf03455390605dead837345c"
format_stamp: "Formatted at 2026-09-25 22:50:15 on dist-test-slave-vdtf"
server_key: "e0eb335d24b91dc420f9ed63c2ec7029"
server_key_iv: "d2b295a0f0fc10aa150c5c4f9557159a"
server_key_version: "encryptionkey@0"
I20260925 22:50:15.162855 22839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 22:50:15.165200 22854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:15.166311 22839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:15.166443 22839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "5bf9cac5cf03455390605dead837345c"
format_stamp: "Formatted at 2026-09-25 22:50:15 on dist-test-slave-vdtf"
server_key: "e0eb335d24b91dc420f9ed63c2ec7029"
server_key_iv: "d2b295a0f0fc10aa150c5c4f9557159a"
server_key_version: "encryptionkey@0"
I20260925 22:50:15.166554 22839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:15.176767 22839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:15.179968 22839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:15.180166 22839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:15.186971 22839 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:42079
I20260925 22:50:15.186990 22906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:42079 every 8 connection(s)
I20260925 22:50:15.187970 22839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:15.190634 22907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:15.195990 22907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c: Bootstrap starting.
I20260925 22:50:15.196430 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 22839
I20260925 22:50:15.196571 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:15.196847 806 external_mini_cluster.cc:1428] Setting key cac119770e9337ee0ad3c749e8c65a03
I20260925 22:50:15.198612 22907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:15.199319 22907 log.cc:824] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:15.201244 22907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c: No bootstrap required, opened a new log
I20260925 22:50:15.204763 22907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf9cac5cf03455390605dead837345c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42079 } }
I20260925 22:50:15.204988 22907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:15.205067 22907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf9cac5cf03455390605dead837345c, State: Initialized, Role: FOLLOWER
I20260925 22:50:15.205475 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:15.198111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48215 (local address 127.0.201.190:42079)
0925 22:50:15.198512 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:15.198527 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:15.198560 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:50:15.199203 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:15.199221 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:15.199604 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:15.200017 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:15.200351 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.201280 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.202174 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.202650 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.202814 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:15.203713 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:15.203720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:15.203740 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:15.204109 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:15.204163 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:15.204185 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:15.204307 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:15.204446 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 22:50:15.204646 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":126,"threads_started":1}
I20260925 22:50:15.205474 22907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf9cac5cf03455390605dead837345c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42079 } }
I20260925 22:50:15.205705 22907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:15.205778 22907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:15.205893 22907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:15.206831 22907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf9cac5cf03455390605dead837345c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42079 } }
I20260925 22:50:15.207176 22907 leader_election.cc:302] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bf9cac5cf03455390605dead837345c; no voters:
I20260925 22:50:15.207512 22907 leader_election.cc:288] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:15.207645 22912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:15.207931 22912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [term 1 LEADER]: Becoming Leader. State: Replica: 5bf9cac5cf03455390605dead837345c, State: Running, Role: LEADER
I20260925 22:50:15.208264 22912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf9cac5cf03455390605dead837345c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42079 } }
I20260925 22:50:15.208369 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:15.208756 22907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:15.209748 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bf9cac5cf03455390605dead837345c. Latest consensus state: current_term: 1 leader_uuid: "5bf9cac5cf03455390605dead837345c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf9cac5cf03455390605dead837345c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42079 } } }
I20260925 22:50:15.209888 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:15.210740 22925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:15.210729 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bf9cac5cf03455390605dead837345c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf9cac5cf03455390605dead837345c" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 42079 } } }
I20260925 22:50:15.210903 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:15.214035 22925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:15.214783 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:15.210598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54071 (local address 127.0.201.190:42079)
0925 22:50:15.210694 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:15.210698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:15.210871 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:50:15.211063 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:15.211066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:15.211108 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:15.211164 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:15.211777 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.212473 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.213282 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.213517 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.213623 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:15.214380 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:15.214383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:15.214386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:15.214398 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:15.214423 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:15.214438 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:15.214511 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:15.214607 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 22:50:15.214653 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:50:15.219282 22925 catalog_manager.cc:1347] Generated new cluster ID: ae9239a9679f4529801ba32064bb0fa4
I20260925 22:50:15.219345 22925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:15.230221 22925 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:15.231372 22925 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:15.248149 22925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bf9cac5cf03455390605dead837345c: Generated new TSK 0
I20260925 22:50:15.248960 22925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:15.252293 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 22839
2026-09-25T22:50:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:15 dist-test-slave-vdtf krb5kdc[22938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:15 dist-test-slave-vdtf krb5kdc[22938](info): set up 2 sockets
Sep 25 22:50:15 dist-test-slave-vdtf krb5kdc[22938](info): commencing operation
krb5kdc: starting...
W20260925 22:50:16.279951 806 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:16 dist-test-slave-vdtf krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:50:16.395937 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:38661
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:38661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:16.503643 22951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:16.503966 22951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:16.504041 22951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:16.508070 22951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:16.508164 22951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:16.508204 22951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:16.508256 22951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:16.508296 22951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:16.513490 22951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:38661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:38661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:16.514521 22951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:16.515544 22951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:16.523495 22959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:16.523509 22956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:16.523559 22957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:16.523801 22951 server_base.cc:1034] running on GCE node
I20260925 22:50:16.524356 22951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:16.525384 22951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:16.526561 22951 hybrid_clock.cc:648] HybridClock initialized: now 1790376616526547 us; error 34 us; skew 500 ppm
Sep 25 22:50:16 dist-test-slave-vdtf krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:16.529235 22951 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:50:16.530082 22951 webserver.cc:460] Webserver started at http://127.0.201.190:42819/ using document root <none> and password file <none>
I20260925 22:50:16.530619 22951 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:16.530702 22951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:16.530916 22951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:16.532915 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "aeda6b7f24824310a9e3b4f3145c751d"
format_stamp: "Formatted at 2026-09-25 22:50:16 on dist-test-slave-vdtf"
server_key: "9ebdc11f3b566cc43b2fc5b13778d012"
server_key_iv: "ec68c2a8d4a0586de1bf606842ec3b00"
server_key_version: "encryptionkey@0"
I20260925 22:50:16.533470 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "aeda6b7f24824310a9e3b4f3145c751d"
format_stamp: "Formatted at 2026-09-25 22:50:16 on dist-test-slave-vdtf"
server_key: "9ebdc11f3b566cc43b2fc5b13778d012"
server_key_iv: "ec68c2a8d4a0586de1bf606842ec3b00"
server_key_version: "encryptionkey@0"
I20260925 22:50:16.537001 22951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:50:16.539258 22966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:16.540429 22951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:16.540561 22951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "aeda6b7f24824310a9e3b4f3145c751d"
format_stamp: "Formatted at 2026-09-25 22:50:16 on dist-test-slave-vdtf"
server_key: "9ebdc11f3b566cc43b2fc5b13778d012"
server_key_iv: "ec68c2a8d4a0586de1bf606842ec3b00"
server_key_version: "encryptionkey@0"
I20260925 22:50:16.540666 22951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:16.545012 22951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:16.547719 22951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:16.547899 22951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:16.554829 22951 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:38661
I20260925 22:50:16.554859 23018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:38661 every 8 connection(s)
I20260925 22:50:16.555785 22951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:16.558569 23019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:16.561103 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 22951
I20260925 22:50:16.561257 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:16.561563 806 external_mini_cluster.cc:1428] Setting key b497eb35117c46ee1105ef9b1d52fa38
I20260925 22:50:16.565093 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d: Bootstrap starting.
I20260925 22:50:16.566995 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:16.562854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35929 (local address 127.0.201.190:38661)
0925 22:50:16.563248 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:16.563263 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:16.563299 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:50:16.564047 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:16.564066 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:16.564367 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:16.564640 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:16.565518 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:16.565523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:16.565539 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:16.565553 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:16.565621 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:16.565651 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:16.565803 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:16.565874 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:50:16.566033 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":135,"threads_started":1}
I20260925 22:50:16.567859 23019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:16.568619 23019 log.cc:824] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:16.569953 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:16.570878 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d: No bootstrap required, opened a new log
I20260925 22:50:16.573933 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:16.572409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48371 (local address 127.0.201.190:38661)
0925 22:50:16.572525 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:16.572529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:16.572679 (+ 150us) server_negotiation.cc:407] Connection header received
0925 22:50:16.572758 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:16.572761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:16.572796 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:16.572884 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:16.573549 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:16.573553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:16.573556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:16.573568 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:16.573601 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:16.573615 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:16.573711 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:16.573764 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 22:50:16.573801 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:50:16.575381 23019 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeda6b7f24824310a9e3b4f3145c751d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38661 } }
I20260925 22:50:16.575620 23019 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:16.575716 23019 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeda6b7f24824310a9e3b4f3145c751d, State: Initialized, Role: FOLLOWER
I20260925 22:50:16.576151 23019 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeda6b7f24824310a9e3b4f3145c751d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38661 } }
I20260925 22:50:16.576308 23019 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:16.576393 23019 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:16.576511 23019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:16.577422 23019 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeda6b7f24824310a9e3b4f3145c751d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38661 } }
I20260925 22:50:16.577768 23019 leader_election.cc:302] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeda6b7f24824310a9e3b4f3145c751d; no voters:
I20260925 22:50:16.578105 23019 leader_election.cc:288] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:16.578208 23029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:16.578447 23029 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [term 1 LEADER]: Becoming Leader. State: Replica: aeda6b7f24824310a9e3b4f3145c751d, State: Running, Role: LEADER
I20260925 22:50:16.578796 23029 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeda6b7f24824310a9e3b4f3145c751d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38661 } }
I20260925 22:50:16.579329 23019 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:16.580297 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeda6b7f24824310a9e3b4f3145c751d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeda6b7f24824310a9e3b4f3145c751d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38661 } } }
I20260925 22:50:16.580425 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:16.580754 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeda6b7f24824310a9e3b4f3145c751d. Latest consensus state: current_term: 1 leader_uuid: "aeda6b7f24824310a9e3b4f3145c751d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeda6b7f24824310a9e3b4f3145c751d" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38661 } } }
I20260925 22:50:16.580825 23038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:16.580835 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:16.583559 23038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:16.589025 23038 catalog_manager.cc:1347] Generated new cluster ID: 2e4ca1465f0a42e689eb9dcee5f07149
I20260925 22:50:16.589088 23038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:16.612653 23038 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:16.613021 23038 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:16.622648 23038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeda6b7f24824310a9e3b4f3145c751d: Generated new TSK 0
I20260925 22:50:16.623342 23038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:16.636318 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 22951
2026-09-25T22:50:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1383 ms)
[----------] 18 tests from AuthTokenIssuingTest (25360 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:16 dist-test-slave-vdtf krb5kdc[23050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:16 dist-test-slave-vdtf krb5kdc[23050](info): set up 2 sockets
Sep 25 22:50:16 dist-test-slave-vdtf krb5kdc[23050](info): commencing operation
krb5kdc: starting...
W20260925 22:50:17.675951 806 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:17 dist-test-slave-vdtf krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:50:17.794255 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:34399
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:34399,127.0.201.189:41521,127.0.201.188:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:17.907851 23063 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:50:17.908190 23063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:17.908252 23063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:17.908326 23063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:17.912470 23063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:17.912551 23063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:17.912590 23063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:17.912643 23063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:17.912683 23063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:17.917965 23063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:34399,127.0.201.189:41521,127.0.201.188:35781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:34399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:17.919124 23063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:17.920212 23063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:17.928422 23071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:17.928438 23068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:17.928474 23063 server_base.cc:1034] running on GCE node
W20260925 22:50:17.928463 23069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:17.929227 23063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:17.930284 23063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:17.931465 23063 hybrid_clock.cc:648] HybridClock initialized: now 1790376617931453 us; error 25 us; skew 500 ppm
Sep 25 22:50:17 dist-test-slave-vdtf krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:17.934198 23063 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:50:17.935011 23063 webserver.cc:460] Webserver started at http://127.0.201.190:45893/ using document root <none> and password file <none>
I20260925 22:50:17.935546 23063 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:17.935631 23063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:17.935877 23063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:17.937845 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "4083ad1a6dc749d49c56709f1b50e47a"
format_stamp: "Formatted at 2026-09-25 22:50:17 on dist-test-slave-vdtf"
server_key: "3aa89f367d3354d1d92d7f5e3e1b5cae"
server_key_iv: "d590ff6f1ebe7e7417b3914fa175235e"
server_key_version: "encryptionkey@0"
I20260925 22:50:17.938392 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "4083ad1a6dc749d49c56709f1b50e47a"
format_stamp: "Formatted at 2026-09-25 22:50:17 on dist-test-slave-vdtf"
server_key: "3aa89f367d3354d1d92d7f5e3e1b5cae"
server_key_iv: "d590ff6f1ebe7e7417b3914fa175235e"
server_key_version: "encryptionkey@0"
I20260925 22:50:17.941926 23063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 22:50:17.944218 23078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:17.945377 23063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:50:17.945504 23063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "4083ad1a6dc749d49c56709f1b50e47a"
format_stamp: "Formatted at 2026-09-25 22:50:17 on dist-test-slave-vdtf"
server_key: "3aa89f367d3354d1d92d7f5e3e1b5cae"
server_key_iv: "d590ff6f1ebe7e7417b3914fa175235e"
server_key_version: "encryptionkey@0"
I20260925 22:50:17.945604 23063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:17.959257 23063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:17.962520 23063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:17.962720 23063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:17.970315 23063 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:34399
I20260925 22:50:17.970580 23130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:34399 every 8 connection(s)
I20260925 22:50:17.971268 23063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:17.974090 23131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:17.977723 23131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:17.979915 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 23063
I20260925 22:50:17.980065 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:17.980381 806 external_mini_cluster.cc:1428] Setting key 1082b51c57197efbf307557414317684
Sep 25 22:50:17 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.201.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.189@KRBTEST.COM" created.
I20260925 22:50:17.993412 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:17.979956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.0.201.190:34399)
0925 22:50:17.980436 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:17.980455 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:17.980718 (+ 263us) server_negotiation.cc:407] Connection header received
0925 22:50:17.981738 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:17.981746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:17.981972 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:17.982168 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:17.982989 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.983940 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.984899 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.985258 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.985419 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:17.987982 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:17.987987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:17.987997 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:17.988018 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:17.989613 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:17.990337 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:17.990345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:17.990346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:17.990525 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:17.990928 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:17.990931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:17.990933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:17.991471 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:17.991599 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:17.992098 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 22:50:17.992428 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":154,"threads_started":1}
I20260925 22:50:17.994076 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:17.979956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:34399 (local address 127.0.0.1:57982)
0925 22:50:17.980549 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:17.980596 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:17.981413 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:17.982176 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:17.982182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:17.982624 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:17.982849 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:17.982861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.984057 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.984060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:17.984782 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:17.984791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.985405 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.985409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:17.985520 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:17.986136 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:17.986152 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:17.987843 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:17.989703 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:17.989716 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:17.989724 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:17.990151 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:17.990643 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:17.990646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:17.990648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:17.990814 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:17.991602 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:17.991609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:17.991803 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 22:50:17.993829 (+ 2026us) client_negotiation.cc:240] Negotiation successful
0925 22:50:17.993962 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"spinlock_wait_cycles":52096,"thread_start_us":154,"threads_started":1}
I20260925 22:50:17.999884 23131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:18.000666 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.000388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:41521 (local address 127.0.0.1:50478)
0925 22:50:18.000522 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.000589 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:41521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 22:50:18.000854 23080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.201.189:41521: connect: Connection refused (error 111)
I20260925 22:50:18.001780 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.001558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:41521 (local address 127.0.0.1:50488)
0925 22:50:18.001656 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.001700 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:41521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 22:50:18.003374 23131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.189:41521: Network error: Client connection negotiation failed: client connection to 127.0.201.189:41521: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab.
Entry for principal kudu/127.0.201.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab.
I20260925 22:50:18.007025 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.189:41521
--webserver_interface=127.0.201.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab
--principal=kudu/127.0.201.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:34399,127.0.201.189:41521,127.0.201.188:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:50:18.052639 23131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } attempt: 1
I20260925 22:50:18.053694 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.053265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:41521 (local address 127.0.0.1:50504)
0925 22:50:18.053498 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.053580 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:41521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 22:50:18.054630 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.054332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:41521 (local address 127.0.0.1:50520)
0925 22:50:18.054486 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.054549 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:41521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:50:18.054761 23131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.189:41521: Network error: Client connection negotiation failed: client connection to 127.0.201.189:41521: connect: Connection refused (error 111)
W20260925 22:50:18.120501 23138 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:50:18.120870 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:18.120939 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:18.121016 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:50:18.122921 23131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } attempt: 2
I20260925 22:50:18.123799 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.123427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:41521 (local address 127.0.0.1:50528)
0925 22:50:18.123597 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.123703 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:41521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 22:50:18.124488 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.124287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:41521 (local address 127.0.0.1:50542)
0925 22:50:18.124384 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.124432 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:41521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 22:50:18.124650 23131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.189:41521: Network error: Client connection negotiation failed: client connection to 127.0.201.189:41521: connect: Connection refused (error 111)
W20260925 22:50:18.125494 23138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:18.125560 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:18.125604 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:18.125640 23138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:18.125681 23138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:18.130960 23138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:34399,127.0.201.189:41521,127.0.201.188:35781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab
--principal=kudu/127.0.201.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.189:41521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.201.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:18.132071 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:18.133066 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:18.141381 23144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:18.141368 23146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:18.141433 23138 server_base.cc:1034] running on GCE node
W20260925 22:50:18.141368 23143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:18.142184 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:18.143241 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:18.144411 23138 hybrid_clock.cc:648] HybridClock initialized: now 1790376618144399 us; error 31 us; skew 500 ppm
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:18.147099 23138 init.cc:377] Logged in from keytab as kudu/127.0.201.189@KRBTEST.COM (short username kudu)
I20260925 22:50:18.147989 23138 webserver.cc:460] Webserver started at http://127.0.201.189:43481/ using document root <none> and password file <none>
I20260925 22:50:18.148598 23138 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:18.148660 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:18.148878 23138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:18.150895 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/data/instance:
uuid: "388206da137448619b65f5f4dd364c10"
format_stamp: "Formatted at 2026-09-25 22:50:18 on dist-test-slave-vdtf"
server_key: "22d2d782e149d8a2dd1a811c4f654e71"
server_key_iv: "d568d1fb9891e2380454077ce4fe7352"
server_key_version: "encryptionkey@0"
I20260925 22:50:18.151428 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/wal/instance:
uuid: "388206da137448619b65f5f4dd364c10"
format_stamp: "Formatted at 2026-09-25 22:50:18 on dist-test-slave-vdtf"
server_key: "22d2d782e149d8a2dd1a811c4f654e71"
server_key_iv: "d568d1fb9891e2380454077ce4fe7352"
server_key_version: "encryptionkey@0"
I20260925 22:50:18.154971 23138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:50:18.157325 23153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:18.158532 23138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:18.158677 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/wal
uuid: "388206da137448619b65f5f4dd364c10"
format_stamp: "Formatted at 2026-09-25 22:50:18 on dist-test-slave-vdtf"
server_key: "22d2d782e149d8a2dd1a811c4f654e71"
server_key_iv: "d568d1fb9891e2380454077ce4fe7352"
server_key_version: "encryptionkey@0"
I20260925 22:50:18.158785 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:18.184057 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:18.187384 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:18.187592 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:18.194373 23138 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.189:41521
I20260925 22:50:18.194643 23205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.189:41521 every 8 connection(s)
I20260925 22:50:18.195345 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
I20260925 22:50:18.198092 23206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:18.201733 23206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:18.203006 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 23138
I20260925 22:50:18.203151 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-1/wal/instance
I20260925 22:50:18.203465 806 external_mini_cluster.cc:1428] Setting key 08f8fda8cb63f288f730ab36654f645b
WARNING: no policy specified for kudu/127.0.201.188@KRBTEST.COM; defaulting to no policy
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.188@KRBTEST.COM" created.
I20260925 22:50:18.215811 23131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } attempt: 3
I20260925 22:50:18.216892 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.203652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.0.201.190:34399)
0925 22:50:18.203817 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.203821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.204776 (+ 955us) server_negotiation.cc:407] Connection header received
0925 22:50:18.206083 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.206085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.206136 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.206217 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.207399 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.208323 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.209443 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.209799 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.209922 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.212416 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.212419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.212421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.212449 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.213997 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.214582 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.214585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.214586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.214658 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.215037 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.215039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.215040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.215219 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.215327 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.216569 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.216696 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 22:50:18.218001 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.204120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:34399 (local address 127.0.0.1:57990)
0925 22:50:18.204619 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.204660 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.205827 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.206432 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.206441 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.206787 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.207244 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.207256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.208450 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.208453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.209324 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.209333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.209942 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.209945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.210086 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.210684 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.210701 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.212250 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:18.214102 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.214107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.214118 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.214485 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.214750 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.214756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.214761 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.214949 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.215342 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:18.215350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:18.215662 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 22:50:18.215906 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 22:50:18.216925 (+ 1019us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":148,"threads_started":1}
I20260925 22:50:18.219508 23206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
I20260925 22:50:18.226766 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.216245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:41521 (local address 127.0.0.1:50544)
0925 22:50:18.216394 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.216412 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.216507 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.217445 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.217448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.217609 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.217727 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.217734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.218983 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.218989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.219661 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.219672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.220554 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.220560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.220691 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.221252 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.221266 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.222783 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:18.225039 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.225044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.225046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.225271 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.225547 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.225550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.225552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.225663 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.226273 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:18.226275 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:18.226352 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 22:50:18.226473 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 22:50:18.226611 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 22:50:18.226950 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.216667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.0.201.189:41521)
0925 22:50:18.217006 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.217019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.217044 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:50:18.217089 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.217098 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.217305 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.217449 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.217861 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.218867 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.219799 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.220401 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.220918 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.222935 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.222946 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.222959 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.222984 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.224933 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.225364 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.225370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.225371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.225450 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.225744 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.225746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.225747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.226159 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.226322 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.226631 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.226776 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":120,"threads_started":1}
I20260925 22:50:18.228612 23131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:18.229313 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.220016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:41521 (local address 127.0.0.1:50548)
0925 22:50:18.220169 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.220184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.220263 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.220594 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.220596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.220762 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.220834 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.220838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.221862 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.221865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.222360 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.222368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.222935 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.222939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.223042 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.223661 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.223707 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.225272 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:18.227903 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.227906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.227908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.228084 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.228394 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.228398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.228400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.228494 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.228862 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:18.228865 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:18.228964 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:50:18.229060 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:50:18.229171 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 22:50:18.229374 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.229117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59622)
0925 22:50:18.229251 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.229303 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 22:50:18.229481 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.220042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.0.201.189:41521)
0925 22:50:18.220423 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.220428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.220436 (+ 8us) server_negotiation.cc:407] Connection header received
0925 22:50:18.220457 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.220459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.220486 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.220584 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.220937 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.221739 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.222472 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.222823 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.222921 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.225409 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.225411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.225413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.225431 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.227731 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.228178 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.228184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.228185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.228292 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.228594 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.228596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.228598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.228780 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.228859 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.229133 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.229268 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":87,"threads_started":1}
I20260925 22:50:18.230214 23206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab.
Entry for principal kudu/127.0.201.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab.
I20260925 22:50:18.230327 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.230049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59624)
0925 22:50:18.230202 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.230267 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260925 22:50:18.230501 23131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:35781: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
I20260925 22:50:18.230823 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.230622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59628)
0925 22:50:18.230707 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.230777 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 22:50:18.231031 23157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
I20260925 22:50:18.231223 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.188:35781
--webserver_interface=127.0.201.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab
--principal=kudu/127.0.201.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:34399,127.0.201.189:41521,127.0.201.188:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:50:18.232003 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.231744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59632)
0925 22:50:18.231870 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.231931 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:50:18.232144 23206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:35781: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
I20260925 22:50:18.261678 23131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } attempt: 1
I20260925 22:50:18.262564 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.262249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59636)
0925 22:50:18.262413 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.262488 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 22:50:18.263278 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.263054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59646)
0925 22:50:18.263159 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.263225 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:50:18.263471 23131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:35781: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
I20260925 22:50:18.281381 23206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } attempt: 1
I20260925 22:50:18.282173 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.281909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59650)
0925 22:50:18.282044 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.282101 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:50:18.282855 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.282655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59658)
0925 22:50:18.282759 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.282805 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:50:18.282974 23206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:35781: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
I20260925 22:50:18.338627 23131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } attempt: 2
W20260925 22:50:18.338626 23214 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:50:18.339017 23214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:18.339074 23214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:18.339121 23214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:50:18.339458 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.339183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59662)
0925 22:50:18.339323 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.339383 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 22:50:18.340225 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.340025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59676)
0925 22:50:18.340119 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.340162 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:50:18.340507 23131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:35781: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
W20260925 22:50:18.343628 23214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:18.343730 23214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:18.343791 23214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:18.343842 23214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:18.343878 23214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:18.349530 23214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:34399,127.0.201.189:41521,127.0.201.188:35781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab
--principal=kudu/127.0.201.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.188:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.201.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:18.350571 23214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:18.351120 23206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } attempt: 2
I20260925 22:50:18.351651 23214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:50:18.351850 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.351539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59678)
0925 22:50:18.351658 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.351755 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 22:50:18.352483 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.352284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59680)
0925 22:50:18.352386 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.352429 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 22:50:18.352600 23206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:35781: Network error: Client connection negotiation failed: client connection to 127.0.201.188:35781: connect: Connection refused (error 111)
W20260925 22:50:18.359699 23220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:18.359778 23222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:18.359714 23219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:18.359882 23214 server_base.cc:1034] running on GCE node
I20260925 22:50:18.360455 23214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:18.361528 23214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:18.362708 23214 hybrid_clock.cc:648] HybridClock initialized: now 1790376618362692 us; error 27 us; skew 500 ppm
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:18.365351 23214 init.cc:377] Logged in from keytab as kudu/127.0.201.188@KRBTEST.COM (short username kudu)
I20260925 22:50:18.366223 23214 webserver.cc:460] Webserver started at http://127.0.201.188:36961/ using document root <none> and password file <none>
I20260925 22:50:18.366758 23214 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:18.366817 23214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:18.367038 23214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:18.369108 23214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/data/instance:
uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f"
format_stamp: "Formatted at 2026-09-25 22:50:18 on dist-test-slave-vdtf"
server_key: "b872ab53137b6f8da44861cffc89e15f"
server_key_iv: "7b49c0e9a6ac131942d622a31b7521c5"
server_key_version: "encryptionkey@0"
I20260925 22:50:18.369652 23214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/wal/instance:
uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f"
format_stamp: "Formatted at 2026-09-25 22:50:18 on dist-test-slave-vdtf"
server_key: "b872ab53137b6f8da44861cffc89e15f"
server_key_iv: "7b49c0e9a6ac131942d622a31b7521c5"
server_key_version: "encryptionkey@0"
I20260925 22:50:18.373209 23214 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:50:18.375414 23229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:18.376654 23214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:50:18.376792 23214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/wal
uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f"
format_stamp: "Formatted at 2026-09-25 22:50:18 on dist-test-slave-vdtf"
server_key: "b872ab53137b6f8da44861cffc89e15f"
server_key_iv: "7b49c0e9a6ac131942d622a31b7521c5"
server_key_version: "encryptionkey@0"
I20260925 22:50:18.376893 23214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:18.405121 23214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:18.408028 23214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:18.408231 23214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:18.415630 23214 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.188:35781
I20260925 22:50:18.415933 23281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.188:35781 every 8 connection(s)
I20260925 22:50:18.416705 23214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
I20260925 22:50:18.419430 23282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:18.423089 23282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:18.426908 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 23214
I20260925 22:50:18.427075 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376341615954-806-0/minicluster-data/master-2/wal/instance
I20260925 22:50:18.427357 806 external_mini_cluster.cc:1428] Setting key 92588179395145a78e624be5d6a3cb75
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
I20260925 22:50:18.437083 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.424931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.0.201.190:34399)
0925 22:50:18.425091 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.425096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.425899 (+ 803us) server_negotiation.cc:407] Connection header received
0925 22:50:18.426756 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.426759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.426820 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.426924 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.428387 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.429097 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.430473 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.430689 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.430826 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.433310 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.433313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.433315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.433339 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.434866 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.435390 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.435393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.435395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.435476 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.435871 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.435875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.435877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.436068 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.436176 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.436805 (+ 629us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.436958 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:50:18.437987 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.425310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:34399 (local address 127.0.0.1:57994)
0925 22:50:18.425749 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.425793 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.426545 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.427131 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.427143 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.427545 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.428200 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.428222 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.429273 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.429277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.430348 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.430357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.430860 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.430863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.431027 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.431602 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.431622 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.433162 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:18.434967 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.434977 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.434986 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.435291 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.435573 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.435576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.435578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.435780 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.436173 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:18.436180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:18.436472 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 22:50:18.436689 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 22:50:18.437003 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":108,"threads_started":1}
I20260925 22:50:18.438535 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.427979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.0.201.188:35781)
0925 22:50:18.428385 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.428396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.428514 (+ 118us) server_negotiation.cc:407] Connection header received
0925 22:50:18.428629 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.428633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.428775 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.428900 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.429813 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.430514 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.431384 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.431917 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.432018 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.434677 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.434683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.434691 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.434711 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.436281 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.436748 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.436754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.436757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.436844 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.437271 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.437274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.437276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.437508 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.437680 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.437891 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.437998 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":256,"threads_started":1}
I20260925 22:50:18.439328 23282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:18.439654 23131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } attempt: 3
I20260925 22:50:18.443717 23206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } attempt: 3
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
I20260925 22:50:18.450766 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.439896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50562 (local address 127.0.201.189:41521)
0925 22:50:18.440022 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.440027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.440040 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:50:18.440181 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.440185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.440231 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.440282 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.440807 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.441541 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.442442 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.442749 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.442844 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.446390 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.446394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.446396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.446423 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.448021 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.449331 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.449336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.449341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.449429 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.449761 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.449765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.449767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.449978 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.450067 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.450425 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.450563 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:50:18.451258 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.439816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:41521 (local address 127.0.0.1:50562)
0925 22:50:18.439965 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.439978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.440046 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.440411 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.440413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.440533 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.440661 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.440667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.441686 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.441691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.442333 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.442341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.442953 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.442957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.443052 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.443547 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.443573 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.445932 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:18.448149 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.448154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.448157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.448448 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.449556 (+ 1108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.449559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.449561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.449651 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.450133 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:18.450136 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:18.450226 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 22:50:18.450914 (+ 688us) client_negotiation.cc:240] Negotiation successful
0925 22:50:18.451038 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":18,"spinlock_wait_cycles":1280}
I20260925 22:50:18.451606 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.440714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.0.201.190:34399)
0925 22:50:18.440813 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.440817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.441044 (+ 227us) server_negotiation.cc:407] Connection header received
0925 22:50:18.441144 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.441147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.441190 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.441254 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.442038 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.442705 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.443617 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.444801 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.445228 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.447378 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.447382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.447384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.447408 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.449113 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.449726 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.449729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.449731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.449813 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.450150 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.450153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.450155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.450386 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.451248 (+ 862us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.451320 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.451444 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":5888}
I20260925 22:50:18.451894 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.440004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59688)
0925 22:50:18.440238 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.440252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.440323 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.440829 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.440832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.441005 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.441108 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.441117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.441778 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.441782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.442526 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.442534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.443061 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.443065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.443148 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.443963 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.443983 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.445591 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:18.450344 (+ 4753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.450347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.450349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.450586 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.450899 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.450902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.450904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.450992 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.451441 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:18.451444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:18.451525 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 22:50:18.451600 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:50:18.451755 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":1}
I20260925 22:50:18.452230 23282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:18.452387 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.440529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.0.201.188:35781)
0925 22:50:18.440628 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.440632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.440645 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:50:18.440679 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.440682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.440719 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.440812 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.441202 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.441670 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.442629 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.442953 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.443046 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.446043 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.446046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.446048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.446083 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.450232 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.450706 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.450709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.450710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.450787 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.451112 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.451115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.451116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.451326 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.452057 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.452132 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.452260 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":3328}
I20260925 22:50:18.455452 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.444082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59702)
0925 22:50:18.444323 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.444337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.444409 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.444862 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.444865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.445047 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.445135 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.445142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.446193 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.446199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.447071 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.447084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.447745 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.447749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.447838 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.448223 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.448243 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.449986 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:18.453918 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.453923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.453925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.454180 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.454466 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.454469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.454472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.454617 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.455000 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:18.455003 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:18.455098 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 22:50:18.455172 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:50:18.455280 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260925 22:50:18.456051 23131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a: Bootstrap starting.
I20260925 22:50:18.456332 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.444309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.0.201.188:35781)
0925 22:50:18.444635 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.444643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.444655 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:50:18.444685 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.444687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.444735 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.444824 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.445320 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.446074 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.447230 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.447616 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.447737 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.450120 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.450124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.450127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.450148 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.453821 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.454280 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.454285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.454287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.454363 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.454722 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.454725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.454727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.454895 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.455906 (+ 1011us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.455977 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.456091 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":104,"threads_started":1}
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
Sep 25 22:50:18 dist-test-slave-vdtf krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
I20260925 22:50:18.459138 23131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:18.461884 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.452700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:35781 (local address 127.0.0.1:59710)
0925 22:50:18.452792 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.452804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.452888 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.454236 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.454240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.454398 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.454490 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.454496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.455403 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.455406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.456199 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.456208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.456756 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.456759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.456871 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.457259 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.457272 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.458777 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:18.460565 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.460568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.460570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.460750 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.461002 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:18.461006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:18.461008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:18.461082 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:18.461408 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:18.461411 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:18.461486 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 22:50:18.461586 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:50:18.461719 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":18}
I20260925 22:50:18.462157 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.453249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.0.201.188:35781)
0925 22:50:18.453992 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.453998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.454010 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:50:18.454038 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.454040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.454079 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.454151 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.454638 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.455304 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.456309 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.456645 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.456759 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.458901 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.458904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.458907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.458927 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.460451 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.460837 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.460843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.460845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.460917 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.461162 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.461168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.461169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.461322 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.461416 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.461635 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.461769 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":673}
I20260925 22:50:18.462929 23206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10: Bootstrap starting.
I20260925 22:50:18.463979 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.453109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.0.201.189:41521)
0925 22:50:18.453385 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.453391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.453405 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:50:18.453439 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.453442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.453480 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.453561 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.454291 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.454980 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.456012 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.456294 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.456388 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.459100 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.459106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.459112 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.459142 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.460787 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.461276 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.461279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.461281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.461363 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.461674 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.461677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.461679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.461891 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.462061 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.463795 (+ 1734us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.463883 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260925 22:50:18.464882 23282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f: Bootstrap starting.
I20260925 22:50:18.465651 23206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:18.466553 23206 log.cc:824] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:18.467036 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:18.467331 23282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:18.467314 23131 log.cc:824] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:18.468241 23282 log.cc:824] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:18.469440 23206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10: No bootstrap required, opened a new log
I20260925 22:50:18.468369 23103 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4083ad1a6dc749d49c56709f1b50e47a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58004
I20260925 22:50:18.470134 23282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f: No bootstrap required, opened a new log
I20260925 22:50:18.472848 23131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a: No bootstrap required, opened a new log
I20260925 22:50:18.473384 23206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } }
I20260925 22:50:18.473613 23206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388206da137448619b65f5f4dd364c10, State: Initialized, Role: FOLLOWER
I20260925 22:50:18.473937 23206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } }
I20260925 22:50:18.473927 23282 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } }
I20260925 22:50:18.474141 23282 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd4efe2c1a03480dba0dc5f33ec8b08f, State: Initialized, Role: FOLLOWER
I20260925 22:50:18.474458 23282 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } }
I20260925 22:50:18.474992 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } } }
I20260925 22:50:18.475294 23206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:18.475284 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:18.475656 23282 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:18.475668 23131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } }
I20260925 22:50:18.475888 23131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4083ad1a6dc749d49c56709f1b50e47a, State: Initialized, Role: FOLLOWER
I20260925 22:50:18.476178 23131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } }
I20260925 22:50:18.476698 23299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } } }
I20260925 22:50:18.476922 23299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:18.477414 23131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:18.477458 23103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:50:18.477545 23103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:18.478868 23103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } }
I20260925 22:50:18.480345 23180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4083ad1a6dc749d49c56709f1b50e47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "388206da137448619b65f5f4dd364c10"
I20260925 22:50:18.480551 23180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [term 0 FOLLOWER]: Advancing to term 1
W20260925 22:50:18.480990 23314 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:50:18.481065 23314 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:50:18.481724 23180 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4083ad1a6dc749d49c56709f1b50e47a in term 1.
I20260925 22:50:18.479269 23214 master_runner.cc:186] Error getting master registration for 127.0.201.190:34399: OK, instance_id { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" instance_seqno: 1790376617968360 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:50:18.483057 23214 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:50:18.483644 23256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4083ad1a6dc749d49c56709f1b50e47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f"
I20260925 22:50:18.483821 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } } }
I20260925 22:50:18.483886 23256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:18.483939 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:18.484920 23256 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4083ad1a6dc749d49c56709f1b50e47a in term 1.
I20260925 22:50:18.479882 23103 leader_election.cc:288] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [CANDIDATE]: Term 1 election: Requested vote from peers 388206da137448619b65f5f4dd364c10 (127.0.201.189:41521), fd4efe2c1a03480dba0dc5f33ec8b08f (127.0.201.188:35781)
I20260925 22:50:18.482117 23080 leader_election.cc:302] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 388206da137448619b65f5f4dd364c10, 4083ad1a6dc749d49c56709f1b50e47a; no voters:
I20260925 22:50:18.486315 23299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:18.486989 23299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [term 1 LEADER]: Becoming Leader. State: Replica: 4083ad1a6dc749d49c56709f1b50e47a, State: Running, Role: LEADER
I20260925 22:50:18.487322 23299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } }
I20260925 22:50:18.489327 23299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4083ad1a6dc749d49c56709f1b50e47a. Latest consensus state: current_term: 1 leader_uuid: "4083ad1a6dc749d49c56709f1b50e47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } } }
I20260925 22:50:18.489476 23299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:18.489884 23335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260925 22:50:18.490867 23315 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:50:18.490957 23315 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:50:18.491142 23138 master_runner.cc:263] No leader master found from master 388206da137448619b65f5f4dd364c10
I20260925 22:50:18.491209 23138 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:50:18.493021 23335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:18.493495 23063 master_runner.cc:263] No leader master found from master 4083ad1a6dc749d49c56709f1b50e47a
I20260925 22:50:18.493563 23063 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:50:18.498661 23256 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [term 1 FOLLOWER]: Refusing update from remote peer 4083ad1a6dc749d49c56709f1b50e47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:18.499337 23299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:18.503113 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4083ad1a6dc749d49c56709f1b50e47a. Latest consensus state: current_term: 1 leader_uuid: "4083ad1a6dc749d49c56709f1b50e47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } } }
I20260925 22:50:18.503257 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:18.504580 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.489456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.0.201.190:34399)
0925 22:50:18.489566 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.489570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.490626 (+ 1056us) server_negotiation.cc:407] Connection header received
0925 22:50:18.491037 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.491043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.491097 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.491156 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.493385 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.494139 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.498164 (+ 4025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.498512 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.498589 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.500805 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.500809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.500812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.500841 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.502656 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.503158 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.503162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.503165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.503263 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.503575 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.503579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.503580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.503814 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.503939 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.504307 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.504428 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:50:18.504980 23180 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [term 1 FOLLOWER]: Refusing update from remote peer 4083ad1a6dc749d49c56709f1b50e47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:18.505537 23299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:18.509328 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4083ad1a6dc749d49c56709f1b50e47a. Latest consensus state: current_term: 1 leader_uuid: "4083ad1a6dc749d49c56709f1b50e47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } } }
I20260925 22:50:18.509477 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:18.512840 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.490142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.0.201.188:35781)
0925 22:50:18.490237 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.490241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.490954 (+ 713us) server_negotiation.cc:407] Connection header received
0925 22:50:18.492356 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.492359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.492401 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.492450 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.495355 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.496125 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.497601 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.497939 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.498018 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.508835 (+ 10817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.508838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.508841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.508869 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.510907 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.511414 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.511418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.511421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.511505 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.511844 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.511851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.511853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.512085 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.512183 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.512549 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.512670 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:50:18.513252 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.489219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.0.201.189:41521)
0925 22:50:18.489589 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.489595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.497683 (+ 8088us) server_negotiation.cc:407] Connection header received
0925 22:50:18.497736 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.497739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.497790 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.497873 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.505536 (+ 7663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.506290 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.507478 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.507853 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.507928 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.509648 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:18.509655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:18.509661 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:18.509696 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:18.511417 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.511892 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.511898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.511901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.511973 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:18.512297 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:18.512301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:18.512302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:18.512531 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:18.512636 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:18.512965 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 22:50:18.513082 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
I20260925 22:50:18.514002 23299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4083ad1a6dc749d49c56709f1b50e47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } } }
I20260925 22:50:18.514127 23299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:18.515688 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4083ad1a6dc749d49c56709f1b50e47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } } }
I20260925 22:50:18.515801 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:18.516163 23340 mvcc.cc:205] Tried to move back new op lower bound from 7333382629361934336 to 7333382629328506880. Current Snapshot: MvccSnapshot[applied={T|T < 7333382629361934336}]
I20260925 22:50:18.517762 23335 catalog_manager.cc:1347] Generated new cluster ID: 4809a3d83f8f4888955366d89dbd1ca4
I20260925 22:50:18.517825 23335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:18.525509 23299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4083ad1a6dc749d49c56709f1b50e47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } } }
I20260925 22:50:18.525635 23299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:18.526906 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4083ad1a6dc749d49c56709f1b50e47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4083ad1a6dc749d49c56709f1b50e47a" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 34399 } } peers { permanent_uuid: "388206da137448619b65f5f4dd364c10" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 41521 } } peers { permanent_uuid: "fd4efe2c1a03480dba0dc5f33ec8b08f" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 35781 } } }
I20260925 22:50:18.527019 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:18.531176 23343 mvcc.cc:205] Tried to move back new op lower bound from 7333382629361934336 to 7333382629328506880. Current Snapshot: MvccSnapshot[applied={T|T < 7333382629361934336}]
I20260925 22:50:18.533401 23335 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:18.536110 23335 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:18.559798 23335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4083ad1a6dc749d49c56709f1b50e47a: Generated new TSK 0
I20260925 22:50:18.560719 23335 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:50:18.594645 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.591296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.0.201.190:34399)
0925 22:50:18.591425 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.591430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.593340 (+ 1910us) server_negotiation.cc:407] Connection header received
0925 22:50:18.593742 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.593746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.593802 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.593859 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.594497 (+ 638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58022: BlockingRecv error: recv got EOF from 127.0.0.1:58022 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:50:18.595777 23286 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.592080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.0.201.188:35781)
0925 22:50:18.592190 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.592196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.593655 (+ 1459us) server_negotiation.cc:407] Connection header received
0925 22:50:18.594995 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.594998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.595047 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.595098 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.595608 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59718: BlockingRecv error: recv got EOF from 127.0.0.1:59718 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 22:50:18.597268 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.592747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.0.201.189:41521)
0925 22:50:18.592859 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.592864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.596250 (+ 3386us) server_negotiation.cc:407] Connection header received
0925 22:50:18.596583 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.596586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.596634 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.596684 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.597143 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50582: BlockingRecv error: recv got EOF from 127.0.0.1:50582 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 22:50:18.601646 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.600242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.0.201.189:41521)
0925 22:50:18.600339 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.600343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.600762 (+ 419us) server_negotiation.cc:407] Connection header received
0925 22:50:18.600966 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.600969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.601039 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.601089 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.601558 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50598: BlockingRecv error: recv got EOF from 127.0.0.1:50598 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 22:50:18.606540 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.605146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.0.201.189:41521)
0925 22:50:18.605259 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.605263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.605727 (+ 464us) server_negotiation.cc:407] Connection header received
0925 22:50:18.605897 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.605900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.605944 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.605992 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.606456 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50606: BlockingRecv error: recv got EOF from 127.0.0.1:50606 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 22:50:18.611905 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.610508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.0.201.189:41521)
0925 22:50:18.610597 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.610601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.611054 (+ 453us) server_negotiation.cc:407] Connection header received
0925 22:50:18.611237 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.611240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.611284 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.611333 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.611814 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50614: BlockingRecv error: recv got EOF from 127.0.0.1:50614 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 22:50:18.619403 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.617922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.0.201.189:41521)
0925 22:50:18.618055 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.618059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.618547 (+ 488us) server_negotiation.cc:407] Connection header received
0925 22:50:18.618723 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.618726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.618773 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.618823 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.619304 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50630: BlockingRecv error: recv got EOF from 127.0.0.1:50630 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 22:50:18.631457 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.629804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.0.201.189:41521)
0925 22:50:18.629925 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.629932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.630490 (+ 558us) server_negotiation.cc:407] Connection header received
0925 22:50:18.630685 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.630688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.630741 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.630795 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.631343 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50644: BlockingRecv error: recv got EOF from 127.0.0.1:50644 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 22:50:18.652180 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.650360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.0.201.189:41521)
0925 22:50:18.650490 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.650494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.651097 (+ 603us) server_negotiation.cc:407] Connection header received
0925 22:50:18.651334 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.651338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.651396 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.651451 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.652060 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50654: BlockingRecv error: recv got EOF from 127.0.0.1:50654 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:50:18.688668 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.686948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.0.201.189:41521)
0925 22:50:18.687073 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.687078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.687665 (+ 587us) server_negotiation.cc:407] Connection header received
0925 22:50:18.687896 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.687899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.687951 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.688002 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.688551 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50660: BlockingRecv error: recv got EOF from 127.0.0.1:50660 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:50:18.757280 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.756044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.0.201.189:41521)
0925 22:50:18.756200 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.756204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.756327 (+ 123us) server_negotiation.cc:407] Connection header received
0925 22:50:18.756452 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.756454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.756500 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.756584 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.757147 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50666: BlockingRecv error: recv got EOF from 127.0.0.1:50666 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:50:18.888518 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.887276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.0.201.189:41521)
0925 22:50:18.887439 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.887443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.887584 (+ 141us) server_negotiation.cc:407] Connection header received
0925 22:50:18.887810 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.887814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.887871 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.887961 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.888409 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50672: BlockingRecv error: recv got EOF from 127.0.0.1:50672 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 22:50:19.147818 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:19.146606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.0.201.189:41521)
0925 22:50:19.146758 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:19.146765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:19.146842 (+ 77us) server_negotiation.cc:407] Connection header received
0925 22:50:19.147021 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:19.147026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:19.147079 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:19.147171 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:19.147625 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50686: BlockingRecv error: recv got EOF from 127.0.0.1:50686 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:50:19.482312 23314 catalog_manager.cc:1259] Loaded cluster ID: 4809a3d83f8f4888955366d89dbd1ca4
I20260925 22:50:19.482427 23314 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f: loading cluster ID for follower catalog manager: success
I20260925 22:50:19.484108 23314 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f: acquiring CA information for follower catalog manager: success
I20260925 22:50:19.485266 23314 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd4efe2c1a03480dba0dc5f33ec8b08f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:50:19.492198 23315 catalog_manager.cc:1259] Loaded cluster ID: 4809a3d83f8f4888955366d89dbd1ca4
I20260925 22:50:19.492273 23315 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10: loading cluster ID for follower catalog manager: success
I20260925 22:50:19.493781 23315 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10: acquiring CA information for follower catalog manager: success
I20260925 22:50:19.494839 23315 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 388206da137448619b65f5f4dd364c10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:50:19.665746 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:19.661947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.0.201.189:41521)
0925 22:50:19.662198 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:19.662203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:19.662332 (+ 129us) server_negotiation.cc:407] Connection header received
0925 22:50:19.662520 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:19.662523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:19.662646 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:19.662798 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:19.663066 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.663577 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.664440 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.664647 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.664731 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.665004 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:19.665425 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:19.665515 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:19.665570 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 22:50:19.665615 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260925 22:50:19.671959 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:19.668590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.0.201.188:35781)
0925 22:50:19.668808 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:19.668812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:19.668827 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:50:19.668901 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:19.668903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:19.669011 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:19.669119 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:19.669359 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.669851 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.670608 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.670826 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.670900 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.671132 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:19.671525 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:19.671613 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:19.671711 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 22:50:19.671773 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":61,"threads_started":1}
I20260925 22:50:19.679308 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:19.674808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.0.201.189:41521)
0925 22:50:19.674941 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:19.674947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:19.675604 (+ 657us) server_negotiation.cc:407] Connection header received
0925 22:50:19.675744 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:19.675746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:19.675803 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:19.675883 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:19.676255 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.676850 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.678180 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.678414 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.678523 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.678825 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:19.678911 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:19.679003 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:19.679119 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 22:50:19.679169 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:50:19.679922 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:19.675224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.0.201.188:35781)
0925 22:50:19.675328 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:19.675332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:19.676334 (+ 1002us) server_negotiation.cc:407] Connection header received
0925 22:50:19.676376 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:19.676378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:19.676467 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:19.676535 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:19.676914 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.677503 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.678650 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.679002 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.679100 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.679318 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:19.679380 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:19.679520 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:19.679742 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 22:50:19.679801 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:50:19.680316 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:19.674789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.0.201.190:34399)
0925 22:50:19.675044 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:19.675049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:19.675408 (+ 359us) server_negotiation.cc:407] Connection header received
0925 22:50:19.675510 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:19.675514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:19.675705 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:19.675822 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:19.676138 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.676986 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.678176 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.678535 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.678795 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.679175 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:19.679498 (+ 323us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:19.679637 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:19.680057 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 22:50:19.680114 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260925 22:50:19.688889 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 23063
I20260925 22:50:19.695050 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 23138
I20260925 22:50:19.700824 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 23214
2026-09-25T22:50:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3066 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:19 dist-test-slave-vdtf krb5kdc[23431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:19 dist-test-slave-vdtf krb5kdc[23431](info): set up 2 sockets
Sep 25 22:50:19 dist-test-slave-vdtf krb5kdc[23431](info): commencing operation
krb5kdc: starting...
W20260925 22:50:20.724099 806 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/test-user.keytab.
Sep 25 22:50:20 dist-test-slave-vdtf krb5kdc[23431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.201.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
Entry for principal kudu/127.0.201.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab.
I20260925 22:50:20.842787 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:38243
--webserver_interface=127.0.201.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:38243,127.0.201.189:44705,127.0.201.188:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:20.952078 23444 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:50:20.952324 23444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:20.952347 23444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:20.952383 23444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:20.956240 23444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:20.956284 23444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:20.956303 23444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:20.956319 23444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:20.956334 23444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:20.961318 23444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:38243,127.0.201.189:44705,127.0.201.188:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.190.keytab
--principal=kudu/127.0.201.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.190:38243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.201.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:20.962229 23444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:20.963140 23444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:20.971267 23450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:20.971318 23452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:20.971350 23444 server_base.cc:1034] running on GCE node
W20260925 22:50:20.971276 23449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:20.972218 23444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:20.973232 23444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:20.974409 23444 hybrid_clock.cc:648] HybridClock initialized: now 1790376620974382 us; error 41 us; skew 500 ppm
Sep 25 22:50:20 dist-test-slave-vdtf krb5kdc[23431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:20.977078 23444 init.cc:377] Logged in from keytab as kudu/127.0.201.190@KRBTEST.COM (short username kudu)
I20260925 22:50:20.977876 23444 webserver.cc:460] Webserver started at http://127.0.201.190:38083/ using document root <none> and password file <none>
I20260925 22:50:20.978425 23444 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:20.978513 23444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:20.978732 23444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:20.980717 23444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/data/instance:
uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda"
format_stamp: "Formatted at 2026-09-25 22:50:20 on dist-test-slave-vdtf"
server_key: "f3906a2437ccf2d3e20b3479000454bc"
server_key_iv: "5d9a2a4dbb8c190040355e1ea62e51f3"
server_key_version: "encryptionkey@0"
I20260925 22:50:20.981269 23444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/wal/instance:
uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda"
format_stamp: "Formatted at 2026-09-25 22:50:20 on dist-test-slave-vdtf"
server_key: "f3906a2437ccf2d3e20b3479000454bc"
server_key_iv: "5d9a2a4dbb8c190040355e1ea62e51f3"
server_key_version: "encryptionkey@0"
I20260925 22:50:20.984705 23444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:50:20.987016 23459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:20.988165 23444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:20.988303 23444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/wal
uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda"
format_stamp: "Formatted at 2026-09-25 22:50:20 on dist-test-slave-vdtf"
server_key: "f3906a2437ccf2d3e20b3479000454bc"
server_key_iv: "5d9a2a4dbb8c190040355e1ea62e51f3"
server_key_version: "encryptionkey@0"
I20260925 22:50:20.988404 23444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:21.010525 23444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:21.013819 23444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:21.014026 23444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:21.020869 23444 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.190:38243
I20260925 22:50:21.021150 23511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.190:38243 every 8 connection(s)
I20260925 22:50:21.021816 23444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:21.024540 23512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:21.028256 23512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:21.028537 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 23444
I20260925 22:50:21.028664 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-0/wal/instance
I20260925 22:50:21.028937 806 external_mini_cluster.cc:1428] Setting key d9ba400e1de6d8f9c8211e532a2e7e96
WARNING: no policy specified for kudu/127.0.201.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.189@KRBTEST.COM" created.
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:50:21.046144 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.030550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:38243 (local address 127.0.0.1:48730)
0925 22:50:21.031132 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.031178 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.032443 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.033211 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.033220 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.033723 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.034118 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.034142 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.035262 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.035266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.036545 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.036560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.037559 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.037563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.037655 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.038264 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.038280 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.040856 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:21.042802 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.042811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.042820 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.043040 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.043395 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.043401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.043405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.043568 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.044421 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:21.044428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:21.044574 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 22:50:21.044739 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 22:50:21.045053 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"spinlock_wait_cycles":75136,"thread_start_us":159,"threads_started":1}
I20260925 22:50:21.046149 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.030550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.0.201.190:38243)
0925 22:50:21.031012 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.031032 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.031308 (+ 276us) server_negotiation.cc:407] Connection header received
0925 22:50:21.032756 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.032760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.032969 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.033158 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.034288 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.035125 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.036667 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.037391 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.037515 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.041002 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.041007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.041016 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.041038 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.042666 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.043150 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.043162 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.043163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.043259 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.043715 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.043720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.043722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.044239 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.044377 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.045115 (+ 738us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.045302 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":134,"threads_started":1}
I20260925 22:50:21.048979 23512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:21.049841 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.049516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:44705 (local address 127.0.0.1:43816)
0925 22:50:21.049670 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.049761 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 22:50:21.050086 23460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.201.189:44705: connect: Connection refused (error 111)
I20260925 22:50:21.051846 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.051580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:44705 (local address 127.0.0.1:43828)
0925 22:50:21.051719 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.051764 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:50:21.051993 23512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.189:44705: Network error: Client connection negotiation failed: client connection to 127.0.201.189:44705: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab.
Entry for principal kudu/127.0.201.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab.
I20260925 22:50:21.058892 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.189:44705
--webserver_interface=127.0.201.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab
--principal=kudu/127.0.201.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:38243,127.0.201.189:44705,127.0.201.188:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:50:21.101250 23512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } attempt: 1
I20260925 22:50:21.102131 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.101817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:44705 (local address 127.0.0.1:43836)
0925 22:50:21.101981 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.102049 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 22:50:21.103003 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.102802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:44705 (local address 127.0.0.1:43840)
0925 22:50:21.102906 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.102950 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:50:21.103240 23512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.189:44705: Network error: Client connection negotiation failed: client connection to 127.0.201.189:44705: connect: Connection refused (error 111)
I20260925 22:50:21.171437 23512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } attempt: 2
I20260925 22:50:21.172447 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.172072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:44705 (local address 127.0.0.1:43844)
0925 22:50:21.172267 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.172347 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 22:50:21.173235 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.173004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:44705 (local address 127.0.0.1:43858)
0925 22:50:21.173126 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.173181 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.189:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:50:21.173369 23512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.189:44705: Network error: Client connection negotiation failed: client connection to 127.0.201.189:44705: connect: Connection refused (error 111)
W20260925 22:50:21.177095 23519 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:50:21.177397 23519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:21.177456 23519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:21.177533 23519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:21.181545 23519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:21.181615 23519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:21.181661 23519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:21.181720 23519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:21.181756 23519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:21.187027 23519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:38243,127.0.201.189:44705,127.0.201.188:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.189.keytab
--principal=kudu/127.0.201.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.189:44705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.201.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:21.188125 23519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:21.189126 23519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:21.197520 23524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:21.197525 23525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:21.197517 23527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:21.197788 23519 server_base.cc:1034] running on GCE node
I20260925 22:50:21.198413 23519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:21.199445 23519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:21.200624 23519 hybrid_clock.cc:648] HybridClock initialized: now 1790376621200605 us; error 41 us; skew 500 ppm
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:21.203300 23519 init.cc:377] Logged in from keytab as kudu/127.0.201.189@KRBTEST.COM (short username kudu)
I20260925 22:50:21.204200 23519 webserver.cc:460] Webserver started at http://127.0.201.189:39931/ using document root <none> and password file <none>
I20260925 22:50:21.204747 23519 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:21.204835 23519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:21.205060 23519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:21.207026 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/data/instance:
uuid: "1d4c49967c584af9a31185b27c594a8a"
format_stamp: "Formatted at 2026-09-25 22:50:21 on dist-test-slave-vdtf"
server_key: "d0111729548bb4549ec5e9246483f1fc"
server_key_iv: "176766ebbbeb9a4e970e82fe24720201"
server_key_version: "encryptionkey@0"
I20260925 22:50:21.207571 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/wal/instance:
uuid: "1d4c49967c584af9a31185b27c594a8a"
format_stamp: "Formatted at 2026-09-25 22:50:21 on dist-test-slave-vdtf"
server_key: "d0111729548bb4549ec5e9246483f1fc"
server_key_iv: "176766ebbbeb9a4e970e82fe24720201"
server_key_version: "encryptionkey@0"
I20260925 22:50:21.211104 23519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:50:21.213492 23534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:21.214655 23519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:21.214798 23519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/wal
uuid: "1d4c49967c584af9a31185b27c594a8a"
format_stamp: "Formatted at 2026-09-25 22:50:21 on dist-test-slave-vdtf"
server_key: "d0111729548bb4549ec5e9246483f1fc"
server_key_iv: "176766ebbbeb9a4e970e82fe24720201"
server_key_version: "encryptionkey@0"
I20260925 22:50:21.214906 23519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:21.233428 23519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:21.236689 23519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:21.236847 23519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:21.243443 23519 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.189:44705
I20260925 22:50:21.243835 23586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.189:44705 every 8 connection(s)
I20260925 22:50:21.244436 23519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/data/info.pb
I20260925 22:50:21.245839 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 23519
I20260925 22:50:21.245991 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-1/wal/instance
I20260925 22:50:21.246286 806 external_mini_cluster.cc:1428] Setting key fa3b3d037ea19e7eb4efc30e4ea9dbd6
I20260925 22:50:21.247323 23587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:21.251053 23587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.201.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.201.188@KRBTEST.COM" created.
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:50:21.264534 23512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } attempt: 3
I20260925 22:50:21.265187 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.253008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.0.201.190:38243)
0925 22:50:21.253184 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.253189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.254034 (+ 845us) server_negotiation.cc:407] Connection header received
0925 22:50:21.254952 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.254955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.255005 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.255083 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.256438 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.256963 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.258087 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.258337 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.258455 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.261229 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.261235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.261240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.261277 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.262971 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.263470 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.263476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.263482 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.263579 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.263974 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.263978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.263981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.264210 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.264316 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.264906 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.265031 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:50:21.266639 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.253338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:38243 (local address 127.0.0.1:48736)
0925 22:50:21.253885 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.253929 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.254763 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.255241 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.255248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.255578 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.256267 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.256291 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.257093 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.257096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.257954 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.257963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.258492 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.258497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.258737 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.259363 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.259389 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.261047 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:21.263098 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.263102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.263114 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.263363 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.263739 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.263741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.263742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.263869 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.264311 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:21.264316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:21.264584 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 22:50:21.264797 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 22:50:21.265560 (+ 763us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":153,"threads_started":1}
I20260925 22:50:21.268028 23587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.201.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab.
Entry for principal kudu/127.0.201.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab.
I20260925 22:50:21.274555 806 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrAtchr/build/debug/bin/kudu
/tmp/dist-test-taskrAtchr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.188:36953
--webserver_interface=127.0.201.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab
--principal=kudu/127.0.201.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.201.148:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.201.190:38243,127.0.201.189:44705,127.0.201.188:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:50:21.275477 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.264986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:44705 (local address 127.0.0.1:43866)
0925 22:50:21.265154 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.265171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.265260 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.265835 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.265838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.266056 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.266164 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.266171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.267313 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.267317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.268252 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.268263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.268919 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.268922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.269050 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.269558 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.269571 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.270822 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:21.273186 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.273190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.273192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.273498 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.274192 (+ 694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.274195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.274197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.274293 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.274848 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:21.274854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:21.274975 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 22:50:21.275065 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:50:21.275223 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:50:21.276295 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.265076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.0.201.189:44705)
0925 22:50:21.265465 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.265483 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.265502 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:50:21.265536 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.265539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.265692 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.265799 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.266280 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.267143 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.268368 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.268743 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.268915 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.270955 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.270964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.270976 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.271006 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.273081 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.273990 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.273996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.273998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.274102 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.274394 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.274401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.274405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.274709 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.275840 (+ 1131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.275967 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.276157 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":89,"threads_started":1}
I20260925 22:50:21.277941 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.268575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:44705 (local address 127.0.0.1:43876)
0925 22:50:21.268681 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.268696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.268798 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.269290 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.269294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.269468 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.269546 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.269551 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.270362 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.270366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.271021 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.271031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.271555 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.271559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.271668 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.272142 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.272160 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.273484 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:21.276506 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.276510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.276514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.276757 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.277042 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.277047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.277048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.277142 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.277553 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:21.277556 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:21.277636 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 22:50:21.277722 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:50:21.277813 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:50:21.278162 23512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:21.278708 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.268760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.0.201.189:44705)
0925 22:50:21.269062 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.269068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.269079 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:50:21.269105 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.269107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.269138 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.269233 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.269655 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.270250 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.271193 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.271428 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.271533 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.273684 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.273687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.273688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.273708 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.276383 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.276856 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.276860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.276862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.276943 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.277240 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.277243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.277245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.277442 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.277542 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.277802 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.277921 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
I20260925 22:50:21.279357 23587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:21.279414 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.279170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60220)
0925 22:50:21.279284 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.279338 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 22:50:21.280022 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.279793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60230)
0925 22:50:21.279880 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.279953 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 22:50:21.280216 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.279881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60236)
0925 22:50:21.280076 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.280136 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260925 22:50:21.280242 23536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
W20260925 22:50:21.280397 23512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:36953: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
I20260925 22:50:21.281186 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.280892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60244)
0925 22:50:21.281044 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.281108 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 22:50:21.281339 23587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:36953: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
I20260925 22:50:21.311579 23512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } attempt: 1
I20260925 22:50:21.312500 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.312158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60252)
0925 22:50:21.312330 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.312404 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 22:50:21.313258 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.313020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60268)
0925 22:50:21.313145 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.313196 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 22:50:21.313405 23512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:36953: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
I20260925 22:50:21.330614 23587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } attempt: 1
I20260925 22:50:21.331439 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.331133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60280)
0925 22:50:21.331284 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.331351 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 22:50:21.332226 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.332009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60282)
0925 22:50:21.332115 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.332164 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:50:21.332407 23587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:36953: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
W20260925 22:50:21.386773 23594 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:50:21.387115 23594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:21.387187 23594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:21.387274 23594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:50:21.388592 23512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } attempt: 2
I20260925 22:50:21.389448 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.389112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60298)
0925 22:50:21.389282 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.389354 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 22:50:21.390192 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.389967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60314)
0925 22:50:21.390080 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.390131 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:50:21.390340 23512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:36953: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
W20260925 22:50:21.391840 23594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:21.391888 23594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:21.391908 23594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:21.391924 23594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:21.391940 23594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:21.397436 23594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.201.148:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.201.190:38243,127.0.201.189:44705,127.0.201.188:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/krb5kdc/kudu_127.0.201.188.keytab
--principal=kudu/127.0.201.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.201.188:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.201.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:50:21.398427 23594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:21.399369 23594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:50:21.400584 23587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } attempt: 2
I20260925 22:50:21.401345 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.401036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60316)
0925 22:50:21.401198 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.401265 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 22:50:21.402163 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.401899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60326)
0925 22:50:21.402022 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.402085 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:50:21.402341 23587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.201.188:36953: Network error: Client connection negotiation failed: client connection to 127.0.201.188:36953: connect: Connection refused (error 111)
W20260925 22:50:21.407955 23603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:21.407955 23601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:21.407948 23600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:21.408246 23594 server_base.cc:1034] running on GCE node
I20260925 22:50:21.408833 23594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:21.409942 23594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:21.411125 23594 hybrid_clock.cc:648] HybridClock initialized: now 1790376621411113 us; error 33 us; skew 500 ppm
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:21.413937 23594 init.cc:377] Logged in from keytab as kudu/127.0.201.188@KRBTEST.COM (short username kudu)
I20260925 22:50:21.414757 23594 webserver.cc:460] Webserver started at http://127.0.201.188:37687/ using document root <none> and password file <none>
I20260925 22:50:21.415266 23594 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:21.415320 23594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:21.415477 23594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:21.417481 23594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/data/instance:
uuid: "899e571a61104ce4bdc7324a6720d881"
format_stamp: "Formatted at 2026-09-25 22:50:21 on dist-test-slave-vdtf"
server_key: "e30ebe26530bad80ce051af2f25a970e"
server_key_iv: "7a6546afcbf9aa68cb9620f09efd571d"
server_key_version: "encryptionkey@0"
I20260925 22:50:21.418037 23594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/wal/instance:
uuid: "899e571a61104ce4bdc7324a6720d881"
format_stamp: "Formatted at 2026-09-25 22:50:21 on dist-test-slave-vdtf"
server_key: "e30ebe26530bad80ce051af2f25a970e"
server_key_iv: "7a6546afcbf9aa68cb9620f09efd571d"
server_key_version: "encryptionkey@0"
I20260925 22:50:21.421696 23594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:50:21.424054 23610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:21.425205 23594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:21.425297 23594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/data,/tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/wal
uuid: "899e571a61104ce4bdc7324a6720d881"
format_stamp: "Formatted at 2026-09-25 22:50:21 on dist-test-slave-vdtf"
server_key: "e30ebe26530bad80ce051af2f25a970e"
server_key_iv: "7a6546afcbf9aa68cb9620f09efd571d"
server_key_version: "encryptionkey@0"
I20260925 22:50:21.425374 23594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:21.437691 23594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:21.440830 23594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:21.440973 23594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:21.447506 23594 rpc_server.cc:307] RPC server started. Bound to: 127.0.201.188:36953
I20260925 22:50:21.447818 23662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.201.188:36953 every 8 connection(s)
I20260925 22:50:21.448496 23594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/data/info.pb
I20260925 22:50:21.451174 23663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:21.452477 806 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrAtchr/build/debug/bin/kudu as pid 23594
I20260925 22:50:21.452633 806 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrAtchr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376341615954-806-0/minicluster-data/master-2/wal/instance
I20260925 22:50:21.452937 806 external_mini_cluster.cc:1428] Setting key c924940c792187aae42f30d8d870bd24
I20260925 22:50:21.455515 23663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:50:21.466960 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.457485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.0.201.190:38243)
0925 22:50:21.457614 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.457619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.457877 (+ 258us) server_negotiation.cc:407] Connection header received
0925 22:50:21.457999 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.458002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.458061 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.458130 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.458918 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.459403 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.460340 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.460553 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.460663 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.463257 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.463261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.463264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.463306 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.464822 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.465327 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.465330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.465332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.465440 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.465809 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.465815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.465818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.466040 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.466142 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.466508 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.466641 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:50:21.467619 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.457353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.190:38243 (local address 127.0.0.1:48748)
0925 22:50:21.457738 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.457788 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.457909 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.458200 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.458206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.458642 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.458794 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.458800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.459524 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.459528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.460208 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.460217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.460699 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.460704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.460812 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.461297 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.461311 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.463094 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:21.464946 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.464954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.464963 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.465236 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.465554 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.465560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.465564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.465716 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.466140 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:21.466145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:21.466263 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 22:50:21.466451 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 22:50:21.466776 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
I20260925 22:50:21.467619 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.454280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.0.201.188:36953)
0925 22:50:21.454696 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.454715 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.454766 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:50:21.455616 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.455640 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.456106 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.456466 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.457137 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.458587 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.459464 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.460063 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.460234 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.462808 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.462813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.462823 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.462844 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.464839 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.465333 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.465340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.465341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.465479 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.465756 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.465758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.465760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.466241 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.466385 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.466661 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.466836 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":275,"thread_start_us":106,"threads_started":1}
I20260925 22:50:21.469276 23663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.190@KRBTEST.COM
I20260925 22:50:21.477960 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.469787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.189:44705 (local address 127.0.0.1:43878)
0925 22:50:21.469908 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.469920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.469979 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.470383 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.470385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.470567 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.470677 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.470683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.471607 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.471610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.472212 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.472221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.472790 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.472793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.472898 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.473323 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.473334 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.474714 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:21.476413 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.476417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.476420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.476649 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.476928 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.476930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.476931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.476999 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.477446 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:21.477452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:21.477553 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 22:50:21.477675 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:50:21.477819 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 22:50:21.478042 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.469923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.0.201.189:44705)
0925 22:50:21.470036 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.470042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.470059 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:50:21.470109 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.470112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.470164 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.470253 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.470806 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.471500 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.472334 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.472680 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.472764 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.474887 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.474890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.474892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.474916 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.476331 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.476741 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.476747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.476750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.476838 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.477104 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.477110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.477113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.477350 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.477443 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.477747 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.477873 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 22:50:21.478704 23663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:50:21.479094 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.470255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.0.201.190:38243)
0925 22:50:21.470365 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.470370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.470383 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:50:21.470438 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.470441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.470483 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.470566 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.471217 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.471775 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.472563 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.472801 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.472898 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.475626 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.475632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.475637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.475665 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.477289 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.477747 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.477751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.477754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.477841 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.478169 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.478174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.478179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.478402 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.478620 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.478778 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.478910 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.188@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.201.189@KRBTEST.COM
I20260925 22:50:21.487022 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.479163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60338)
0925 22:50:21.479420 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.479433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.479518 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.479771 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.479774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.479913 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.479978 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.479983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.480747 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.480753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.481269 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.481275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.481713 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.481716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.481786 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.482212 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.482221 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.483607 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:21.485357 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.485362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.485367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.485609 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.485884 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.485889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.485892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.485988 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.486436 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:21.486439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:21.486544 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:50:21.486686 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 22:50:21.486858 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260925 22:50:21.487048 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.479193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.0.201.188:36953)
0925 22:50:21.479305 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.479308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.479500 (+ 192us) server_negotiation.cc:407] Connection header received
0925 22:50:21.479594 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.479596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.479637 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.479756 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.480066 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.480631 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.481376 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.481609 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.481717 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.483767 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.483772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.483776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.483809 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.485251 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.485700 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.485704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.485705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.485785 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.486087 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.486090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.486092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.486331 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.486426 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.486747 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.486920 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":53}
I20260925 22:50:21.489141 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.480251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.0.201.189:44705)
0925 22:50:21.480363 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.480367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.480379 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:50:21.480500 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.480503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.480540 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.480624 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.481175 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.481634 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.482537 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.482878 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.482997 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.485561 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.485564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.485565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.485586 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.487372 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.487883 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.487887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.487888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.487980 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.488353 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.488356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.488357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.488577 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.488764 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.488862 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.488973 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:50:21.489486 23512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } attempt: 3
I20260925 22:50:21.490912 23663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881: Bootstrap starting.
I20260925 22:50:21.493497 23587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } attempt: 3
I20260925 22:50:21.493696 23663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:21.494480 806 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:50:21.494561 23663 log.cc:824] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.190@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
I20260925 22:50:21.495627 23486 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48760
I20260925 22:50:21.498351 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.489904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60346)
0925 22:50:21.490014 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.490030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.490105 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.490454 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.490457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.490617 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.490690 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.490695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.491430 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.491435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.492218 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.492230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.492785 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.492788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.492871 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.493309 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.493323 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.495109 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:21.496839 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.496843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.496846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.497043 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.497320 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.497324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.497326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.497415 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.497849 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:21.497852 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:21.497962 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 22:50:21.498054 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:50:21.498184 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 22:50:21.498564 23663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881: No bootstrap required, opened a new log
I20260925 22:50:21.498579 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.490164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.0.201.188:36953)
0925 22:50:21.490283 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.490286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.490295 (+ 9us) server_negotiation.cc:407] Connection header received
0925 22:50:21.490320 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.490322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.490351 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.490434 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.490804 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.491326 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.492312 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.492593 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.492784 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.495274 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.495278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.495280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.495305 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.496722 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.497139 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.497144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.497146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.497224 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.497509 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.497512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.497514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.497753 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.497844 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.498165 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.498321 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 22:50:21 dist-test-slave-vdtf krb5kdc[23431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.201.189@KRBTEST.COM for kudu/127.0.201.188@KRBTEST.COM
I20260925 22:50:21.501543 23512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda: Bootstrap starting.
I20260925 22:50:21.502935 23663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } }
I20260925 22:50:21.503175 23663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899e571a61104ce4bdc7324a6720d881, State: Initialized, Role: FOLLOWER
I20260925 22:50:21.503502 23663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } }
I20260925 22:50:21.504341 23670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } } }
I20260925 22:50:21.504478 23512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:21.504487 23670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:21.504621 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.494033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.201.188:36953 (local address 127.0.0.1:60352)
0925 22:50:21.494200 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.494214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.494292 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.495736 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.495740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.495909 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.495996 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.496002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.497129 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.497132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.497894 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.497904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.498929 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.498932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.499014 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.499455 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.499469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.500865 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:21.503062 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.503065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.503067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.503316 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.503618 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:21.503621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:21.503623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:21.503743 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:21.504155 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:21.504158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:21.504257 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:50:21.504350 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 22:50:21.504476 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 22:50:21.504855 23663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:21.505151 23512 log.cc:824] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:21.505455 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.494065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.0.201.188:36953)
0925 22:50:21.495483 (+ 1418us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.495491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.495504 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:50:21.495537 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.495539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.495577 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.496102 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.496210 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.496932 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.498032 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.498370 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.499215 (+ 845us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.501006 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.501009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.501012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.501034 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.502935 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.503437 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.503440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.503442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.503521 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.503856 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.503860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.503861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.504057 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.504154 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.505150 (+ 996us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.505280 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1358,"thread_start_us":98,"threads_started":1}
I20260925 22:50:21.507570 23512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda: No bootstrap required, opened a new log
I20260925 22:50:21.508330 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a: Bootstrap starting.
W20260925 22:50:21.510257 23682 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:50:21.510325 23682 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:50:21.510505 23512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } }
I20260925 22:50:21.510743 23512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9539cfa3b7441ba6eb5d5c12af8bda, State: Initialized, Role: FOLLOWER
I20260925 22:50:21.510813 23587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:21.511089 23512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } }
I20260925 22:50:21.511615 23684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } } }
I20260925 22:50:21.511909 23684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:21.512351 23512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:21.511662 23587 log.cc:824] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:21.514401 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a: No bootstrap required, opened a new log
W20260925 22:50:21.515602 23465 tablet.cc:2369] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:50:21.512527 23594 master_runner.cc:186] Error getting master registration for 127.0.201.190:38243: OK, instance_id { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" instance_seqno: 1790376621018951 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:50:21.515792 23594 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:50:21.516011 23486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:50:21.516121 23486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:21.518496 23587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } }
I20260925 22:50:21.518713 23587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4c49967c584af9a31185b27c594a8a, State: Initialized, Role: FOLLOWER
I20260925 22:50:21.519043 23587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } }
I20260925 22:50:21.519522 23697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } } }
I20260925 22:50:21.519758 23697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:21.520419 23587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:21.517408 23444 master_runner.cc:186] Error getting master registration for 127.0.201.189:44705: OK, instance_id { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" instance_seqno: 1790376621241677 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:50:21.521351 23444 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:50:21.517416 23486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } }
I20260925 22:50:21.522449 23486 leader_election.cc:288] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [CANDIDATE]: Term 1 election: Requested vote from peers 1d4c49967c584af9a31185b27c594a8a (127.0.201.189:44705), 899e571a61104ce4bdc7324a6720d881 (127.0.201.188:36953)
I20260925 22:50:21.523262 23637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "899e571a61104ce4bdc7324a6720d881"
I20260925 22:50:21.523488 23637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:21.523428 23561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1d4c49967c584af9a31185b27c594a8a"
I20260925 22:50:21.524817 23561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:21.524869 23637 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9539cfa3b7441ba6eb5d5c12af8bda in term 1.
I20260925 22:50:21.525281 23461 leader_election.cc:302] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9539cfa3b7441ba6eb5d5c12af8bda, 899e571a61104ce4bdc7324a6720d881; no voters:
I20260925 22:50:21.525544 23684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:21.525753 23684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [term 1 LEADER]: Becoming Leader. State: Replica: 7e9539cfa3b7441ba6eb5d5c12af8bda, State: Running, Role: LEADER
W20260925 22:50:21.525945 23708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:50:21.526005 23708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:50:21.526078 23684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } }
I20260925 22:50:21.526134 23561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9539cfa3b7441ba6eb5d5c12af8bda in term 1.
I20260925 22:50:21.527815 23684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9539cfa3b7441ba6eb5d5c12af8bda. Latest consensus state: current_term: 1 leader_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } } }
I20260925 22:50:21.527923 23684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [sys.catalog]: This master's current role is: LEADER
W20260925 22:50:21.531055 23696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:50:21.531167 23696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:50:21.532200 23717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:21.533648 23717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:21.537603 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.527240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.0.201.188:36953)
0925 22:50:21.527358 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.527363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.527744 (+ 381us) server_negotiation.cc:407] Connection header received
0925 22:50:21.527947 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.527951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.528001 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.528053 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.529194 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.530047 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.531333 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.531723 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.531807 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.533721 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.533725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.533727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.533755 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.535606 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.536150 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.536154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.536156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.536258 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.536577 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.536580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.536583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.536802 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.536919 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.537316 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.537439 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 22:50:21.547077 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.532037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.0.201.190:38243)
0925 22:50:21.532142 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.532146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.538178 (+ 6032us) server_negotiation.cc:407] Connection header received
0925 22:50:21.538373 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.538376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.538424 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.538473 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.539333 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.540081 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.541224 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.541561 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.541631 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.543273 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.543277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.543279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.543304 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.545166 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.545702 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.545707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.545710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.545806 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.546103 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.546107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.546109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.546325 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.546429 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.546789 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.546905 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:50:21.551601 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.533427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.0.201.189:44705)
0925 22:50:21.533542 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.533546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.539926 (+ 6380us) server_negotiation.cc:407] Connection header received
0925 22:50:21.540155 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.540161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.540215 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.540266 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.541153 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.541866 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.543084 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.543396 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.543478 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.547833 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:21.547837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:21.547840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:21.547866 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:21.549805 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.550288 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.550292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.550294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.550379 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:21.550665 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:21.550669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:21.550671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:21.550875 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:21.550976 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:21.551322 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 22:50:21.551438 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:50:21.552137 23637 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [term 1 FOLLOWER]: Refusing update from remote peer 7e9539cfa3b7441ba6eb5d5c12af8bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:21.552846 23714 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:21.557039 23670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9539cfa3b7441ba6eb5d5c12af8bda. Latest consensus state: current_term: 1 leader_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } } }
I20260925 22:50:21.557173 23670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:21.558108 23561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [term 1 FOLLOWER]: Refusing update from remote peer 7e9539cfa3b7441ba6eb5d5c12af8bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:21.559849 23714 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:21.563463 23697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9539cfa3b7441ba6eb5d5c12af8bda. Latest consensus state: current_term: 1 leader_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } } }
I20260925 22:50:21.563589 23697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:21.566345 23714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } } }
I20260925 22:50:21.566465 23714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:21.568948 23697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } } }
I20260925 22:50:21.569083 23697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:21.569794 23721 mvcc.cc:205] Tried to move back new op lower bound from 7333382641868959744 to 7333382641773842432. Current Snapshot: MvccSnapshot[applied={T|T < 7333382641868959744}]
I20260925 22:50:21.572042 23724 mvcc.cc:205] Tried to move back new op lower bound from 7333382641868959744 to 7333382641773842432. Current Snapshot: MvccSnapshot[applied={T|T < 7333382641868959744}]
I20260925 22:50:21.574033 23684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } } }
I20260925 22:50:21.574153 23684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:21.575464 23722 mvcc.cc:205] Tried to move back new op lower bound from 7333382641868959744 to 7333382641773842432. Current Snapshot: MvccSnapshot[applied={T|T < 7333382641868959744}]
I20260925 22:50:21.575594 23670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9539cfa3b7441ba6eb5d5c12af8bda" member_type: VOTER last_known_addr { host: "127.0.201.190" port: 38243 } } peers { permanent_uuid: "1d4c49967c584af9a31185b27c594a8a" member_type: VOTER last_known_addr { host: "127.0.201.189" port: 44705 } } peers { permanent_uuid: "899e571a61104ce4bdc7324a6720d881" member_type: VOTER last_known_addr { host: "127.0.201.188" port: 36953 } } }
I20260925 22:50:21.575692 23670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:21.577253 23717 catalog_manager.cc:1347] Generated new cluster ID: d8068f2cab954974b5a522dd511d7ed4
I20260925 22:50:21.577308 23717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:21.607858 23717 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:21.609226 23717 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:21.625247 23717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e9539cfa3b7441ba6eb5d5c12af8bda: Generated new TSK 0
I20260925 22:50:21.628227 23717 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:50:21.667302 23668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.664293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.0.201.188:36953)
0925 22:50:21.664437 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.664442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.666196 (+ 1754us) server_negotiation.cc:407] Connection header received
0925 22:50:21.666386 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.666390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.666438 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.666492 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.667149 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60366: BlockingRecv error: recv got EOF from 127.0.0.1:60366 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 22:50:21.668902 23515 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.665019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.0.201.190:38243)
0925 22:50:21.665136 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.665140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.668029 (+ 2889us) server_negotiation.cc:407] Connection header received
0925 22:50:21.668217 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.668220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.668269 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.668320 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.668797 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48772: BlockingRecv error: recv got EOF from 127.0.0.1:48772 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 22:50:21.670483 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.665653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.0.201.189:44705)
0925 22:50:21.665764 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.665768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.669553 (+ 3785us) server_negotiation.cc:407] Connection header received
0925 22:50:21.669721 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.669725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.669778 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.669829 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.670335 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43906: BlockingRecv error: recv got EOF from 127.0.0.1:43906 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 22:50:21.674674 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.673425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.0.201.189:44705)
0925 22:50:21.673561 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.673566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.673783 (+ 217us) server_negotiation.cc:407] Connection header received
0925 22:50:21.673996 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.673999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.674070 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.674119 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.674577 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43922: BlockingRecv error: recv got EOF from 127.0.0.1:43922 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 22:50:21.679652 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.678216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.0.201.189:44705)
0925 22:50:21.678327 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.678330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.678801 (+ 471us) server_negotiation.cc:407] Connection header received
0925 22:50:21.678973 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.678976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.679027 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.679088 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.679559 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43928: BlockingRecv error: recv got EOF from 127.0.0.1:43928 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:50:21.685400 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.683958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.0.201.189:44705)
0925 22:50:21.684055 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.684059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.684508 (+ 449us) server_negotiation.cc:407] Connection header received
0925 22:50:21.684696 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.684699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.684749 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.684814 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.685302 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43940: BlockingRecv error: recv got EOF from 127.0.0.1:43940 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 22:50:21.693382 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.691801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.0.201.189:44705)
0925 22:50:21.691931 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.691935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.692407 (+ 472us) server_negotiation.cc:407] Connection header received
0925 22:50:21.692663 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.692666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.692716 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.692776 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.693288 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43956: BlockingRecv error: recv got EOF from 127.0.0.1:43956 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 22:50:21.705113 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.703663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.0.201.189:44705)
0925 22:50:21.703823 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.703827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.704289 (+ 462us) server_negotiation.cc:407] Connection header received
0925 22:50:21.704457 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.704460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.704504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.704551 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.705017 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43968: BlockingRecv error: recv got EOF from 127.0.0.1:43968 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260925 22:50:21.728976 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.723726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.0.201.189:44705)
0925 22:50:21.723836 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.723840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.728002 (+ 4162us) server_negotiation.cc:407] Connection header received
0925 22:50:21.728228 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.728231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.728307 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.728389 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.728879 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43978: BlockingRecv error: recv got EOF from 127.0.0.1:43978 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 22:50:21.764740 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.763507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.0.201.189:44705)
0925 22:50:21.763666 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.763696 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.763912 (+ 216us) server_negotiation.cc:407] Connection header received
0925 22:50:21.764019 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.764021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.764069 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.764142 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.764611 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43994: BlockingRecv error: recv got EOF from 127.0.0.1:43994 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 22:50:21.831916 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.830785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.0.201.189:44705)
0925 22:50:21.830943 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.830950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.831090 (+ 140us) server_negotiation.cc:407] Connection header received
0925 22:50:21.831218 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.831220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.831267 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.831340 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.831792 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44006: BlockingRecv error: recv got EOF from 127.0.0.1:44006 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 22:50:21.963253 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.961986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.0.201.189:44705)
0925 22:50:21.962157 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.962162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.962301 (+ 139us) server_negotiation.cc:407] Connection header received
0925 22:50:21.962482 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.962485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.962532 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.962624 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.963119 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44014: BlockingRecv error: recv got EOF from 127.0.0.1:44014 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 22:50:22.222604 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:22.221402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.0.201.189:44705)
0925 22:50:22.221541 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:22.221546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:22.221664 (+ 118us) server_negotiation.cc:407] Connection header received
0925 22:50:22.221840 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:22.221843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:22.221902 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:22.221991 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:22.222482 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44016: BlockingRecv error: recv got EOF from 127.0.0.1:44016 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:50:22.511818 23682 catalog_manager.cc:1259] Loaded cluster ID: d8068f2cab954974b5a522dd511d7ed4
I20260925 22:50:22.511937 23682 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881: loading cluster ID for follower catalog manager: success
I20260925 22:50:22.513799 23682 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881: acquiring CA information for follower catalog manager: success
I20260925 22:50:22.515009 23682 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 899e571a61104ce4bdc7324a6720d881: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:50:22.527093 23708 catalog_manager.cc:1259] Loaded cluster ID: d8068f2cab954974b5a522dd511d7ed4
I20260925 22:50:22.527158 23708 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a: loading cluster ID for follower catalog manager: success
I20260925 22:50:22.528873 23708 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a: acquiring CA information for follower catalog manager: success
I20260925 22:50:22.530022 23708 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d4c49967c584af9a31185b27c594a8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:50:22.740476 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:22.736710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.0.201.189:44705)
0925 22:50:22.736944 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:22.736947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:22.736985 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:50:22.737125 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:22.737128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:22.737251 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:22.737402 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:22.737701 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.738229 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.739051 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.739271 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.739345 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.739618 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:22.740077 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:22.740167 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:22.740302 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 22:50:22.740351 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260925 22:50:22.746906 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:22.743219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.0.201.188:36953)
0925 22:50:22.743466 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:22.743471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:22.743485 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:50:22.743627 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:22.743630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:22.743762 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:22.743899 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:22.744120 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.744628 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.745456 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.745684 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.745799 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.746090 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:22.746502 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:22.746637 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:22.746677 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 22:50:22.746731 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260925 22:50:22.754290 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:22.750113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.0.201.189:44705)
0925 22:50:22.750216 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:22.750220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:22.750548 (+ 328us) server_negotiation.cc:407] Connection header received
0925 22:50:22.750701 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:22.750704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:22.750756 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:22.750854 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:22.751080 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.751812 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.752976 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.753304 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.753381 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.753783 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:22.753867 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:22.753933 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:22.754079 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 22:50:22.754141 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 22:50:22.754680 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:22.749780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.0.201.188:36953)
0925 22:50:22.750083 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:22.750088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:22.750215 (+ 127us) server_negotiation.cc:407] Connection header received
0925 22:50:22.750367 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:22.750372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:22.750419 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:22.750490 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:22.750776 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.751377 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.752887 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.753214 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.753318 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.753789 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:22.753870 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:22.753952 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:22.754503 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 22:50:22.754560 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260925 22:50:22.754770 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:22.749574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.0.201.190:38243)
0925 22:50:22.749971 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:22.749976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:22.749993 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:50:22.750091 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:22.750094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:22.750260 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:22.750390 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:22.750621 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.751433 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.752887 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.753236 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.753311 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.753899 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:22.754256 (+ 357us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:22.754351 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:22.754556 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 22:50:22.754624 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":65,"threads_started":1}
I20260925 22:50:22.758071 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 23444
I20260925 22:50:22.764364 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 23519
I20260925 22:50:22.769862 806 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrAtchr/build/debug/bin/kudu with pid 23594
2026-09-25T22:50:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3069 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6135 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281147 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST